I used around 200 Gb of storage

kRxZy_kRxZy wrote:

I used Python with about 100 concurrent threads to fetch data from the API efficiently. The dataset contained roughly 150 million users grinding my storage, and I was collecting all api data. I stored all data in a ndjson file.

Where did you manage to get a dataset of 150 MILLION scratch users? I'm using SUI which only has 9 million

redspacecat wrote:

kRxZy_kRxZy wrote:

I used Python with about 100 concurrent threads to fetch data from the API efficiently. The dataset contained roughly 150 million users grinding my storage, and I was collecting all api data. I stored all data in a ndjson file.

Where did you manage to get a dataset of 150 MILLION scratch users? I'm using SUI which only has 9 million

It's around 150 million https://annualreport.scratchfoundation.org/

kRxZy_kRxZy wrote:

redspacecat wrote:

kRxZy_kRxZy wrote:

I used Python with about 100 concurrent threads to fetch data from the API efficiently. The dataset contained roughly 150 million users grinding my storage, and I was collecting all api data. I stored all data in a ndjson file.

Where did you manage to get a dataset of 150 MILLION scratch users? I'm using SUI which only has 9 million

It's around 150 million https://annualreport.scratchfoundation.org/

That doesn't have a list of usernames, does it?

redspacecat wrote:

kRxZy_kRxZy wrote:

redspacecat wrote:

kRxZy_kRxZy wrote:

I used Python with about 100 concurrent threads to fetch data from the API efficiently. The dataset contained roughly 150 million users grinding my storage, and I was collecting all api data. I stored all data in a ndjson file.

Where did you manage to get a dataset of 150 MILLION scratch users? I'm using SUI which only has 9 million

It's around 150 million https://annualreport.scratchfoundation.org/

That doesn't have a list of usernames, does it?

i dont think it does

kRxZy_kRxZy wrote:

redspacecat wrote:

kRxZy_kRxZy wrote:

redspacecat wrote:

kRxZy_kRxZy wrote:

I used Python with about 100 concurrent threads to fetch data from the API efficiently. The dataset contained roughly 150 million users grinding my storage, and I was collecting all api data. I stored all data in a ndjson file.

Where did you manage to get a dataset of 150 MILLION scratch users? I'm using SUI which only has 9 million

It's around 150 million https://annualreport.scratchfoundation.org/

That doesn't have a list of usernames, does it?

i dont think it does

Then what were you fetching from the api?

redspacecat wrote:

kRxZy_kRxZy wrote:

redspacecat wrote:

kRxZy_kRxZy wrote:

redspacecat wrote:

kRxZy_kRxZy wrote:

I used Python with about 100 concurrent threads to fetch data from the API efficiently. The dataset contained roughly 150 million users grinding my storage, and I was collecting all api data. I stored all data in a ndjson file.

Where did you manage to get a dataset of 150 MILLION scratch users? I'm using SUI which only has 9 million

It's around 150 million https://annualreport.scratchfoundation.org/

That doesn't have a list of usernames, does it?

i dont think it does

Then what were you fetching from the api?

I had a script with an auto name generator, I started with aa, then ab, then ac and so on

BIG UPDATE!

acohen_bhusd wrote:

please add one link to GET someone’s exact post count

You can now see the post count and history of users!


The website has also been updated to look better.

There are also now forum leaderboards page that shows you rankings of users on the forums.
https://scratchinfo.quuq.dev/forums

Magic Crayon is at the top of Advanced Topics lol

redspacecat wrote:

BIG UPDATE!

acohen_bhusd wrote:

please add one link to GET someone’s exact post count

You can now see the post count and history of users!


The website has also been updated to look better.

There are also now forum leaderboards page that shows you rankings of users on the forums.
https://scratchinfo.quuq.dev/forums

How did you get that data? Did your just count every post?

Are you scraping data cos, wow. Ate you scraping forum data or user data or both?

It looks correct mostly (before posting this [so this counter excludes this post], Scratch says I have 22548 posts, and you can check what Scratch Info says there)
My browser / operating system: Windows Vista, Firefox 145.0, No Flash versions detected
Just wondering, but were topics/posts moved or sth, or does this just not include them? I remember posting in some forums not listed there lol (in the other language forums) but I could be wrong

medians wrote:

It looks correct mostly (before posting this [so this counter excludes this post], Scratch says I have 22548 posts, and you can check what Scratch Info says there)
My browser / operating system: Windows Vista, Firefox 145.0, No Flash versions detected
Just wondering, but were topics/posts moved or sth, or does this just not include them? I remember posting in some forums not listed there lol (in the other language forums) but I could be wrong

I don't know if moved/merged posts do anything.

This isn't entirely accurate as some posted were skipped and there were some bugs. If there are certain post ids you see that aren't included, I can add them manually.

minniesworld wrote:

How did you get that data? Did your just count every post?

Yes, but I don't need to count every post because there are multiple posts per topic page, so I don't actually need to make 9 million requests.

kRxZy_kRxZy wrote:

Are you scraping data cos, wow. Ate you scraping forum data or user data or both?

For this I'm only scraping when the post was created, the author, and what subforum it's in.
This was a forums only thing though, if that's what you mean.

redspacecat wrote:

medians wrote:

It looks correct mostly (before posting this [so this counter excludes this post], Scratch says I have 22548 posts, and you can check what Scratch Info says there)
My browser / operating system: Windows Vista, Firefox 145.0, No Flash versions detected
Just wondering, but were topics/posts moved or sth, or does this just not include them? I remember posting in some forums not listed there lol (in the other language forums) but I could be wrong

I don't know if moved/merged posts do anything.

This isn't entirely accurate as some posted were skipped and there were some bugs. If there are certain post ids you see that aren't included, I can add them manually.

Yeah I don't know how easy that would be when I have more than 20000 posts lol
Edit: Also by “other language forums”, I mean languages that aren't English

medians wrote:

Yeah I don't know how easy that would be when I have more than 20000 posts lol
Edit: Also by “other language forums”, I mean languages that aren't English

Sorry I'm not sure if there's much I can do there. I can try to look over the data and check for stuff.

bump

bump

For some reason I wonder about word counts (like how many times I've said Scratch 2.0 on the forums) LOL

wow i adore the forum leaderboard thing! Love it so so so so so so so so much! Never knew my old account was ranked just below 700 in posts

my current account is ranked 6951 on the ledarboard but since the post data is inaccurate (I have 192 posts as of writing this one) I’m actually ranked anywhere between 5336 to 5359