SUI v3.0 - Scratch Username Index API [3.7M+ Indexed Users!]

DifferentDance8

Sid72020123 wrote:
SUI indexed more than 3.7M+ users!

But the indexing amount seems less. Any more ways to index more users?

That's a lot of users. Unfortunately, that's only like 5% of the 110,051,160 users registered

Words? scoffs Who needs one? not when you have blinkies stamps - so confusing terminologies!

Sid72020123

DifferentDance8 wrote:
(#221)
Sid72020123 wrote:
SUI indexed more than 3.7M+ users!

But the indexing amount seems less. Any more ways to index more users?
That's a lot of users. Unfortunately, that's only like 5% of the 110,051,160 users registered

Yeah. I know that. Indexing usernames on Scratch is hard. I wonder how ScratchDB finds new users. I have limited resources to run my server.

But my program almost upserts (updates + inserts) 600,000 users every day!

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

ajskateboarder

Sid72020123 wrote:
DifferentDance8 wrote:
That's a lot of users. Unfortunately, that's only like 5% of the 110,051,160 users registered
Yeah. I know that. Indexing usernames on Scratch is hard. I wonder how ScratchDB finds new users. I have limited resources to run my server.

But my program almost upserts (updates + inserts) 600,000 users every day!

You could expedite the process by running multiple instances of SUI at once. What are your server's specs?

I always seemed to get mad at people in 2022 posts and overuse things like :facepalm: If you are looking at one of those posts, just note I don't write like this anymore
software developer
artist
swimming and biking enjoyer

Sid72020123

ajskateboarder wrote:
(#223)
What are your server's specs?

Hmm… I'm using a free hosting service but here is a small info about it:

ajskateboarder wrote:
(#223)
You could expedite the process by running multiple instances of SUI at once.

I can do that. But don't you think that it will over-spam the Scratch API? Also, there are already 3 threads running:

Username Indexer - It indexes the followers and following of famous Scratchers…
Project Indexer - It indexes the project author and the author's followers…
Studio Indexer - It indexes the managers and curators in a studio…

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

ajskateboarder

Sid72020123 wrote:
I can do that. But don't you think that it will over-spam the Scratch API?

You could make it so the additional threads index at a slower rate, like 1 request per second, while the main threads run as usual.

I always seemed to get mad at people in 2022 posts and overuse things like :facepalm: If you are looking at one of those posts, just note I don't write like this anymore
software developer
artist
swimming and biking enjoyer

Sid72020123

ajskateboarder wrote:
(#225)
Sid72020123 wrote:
I can do that. But don't you think that it will over-spam the Scratch API?
You could make it so the additional threads index at a slower rate, like 1 request per second, while the main threads run as usual.

Hmm… Maybe I can make that. I will try that soon…
Thank you for the suggestion and idea!

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

Sid72020123

I am going to re-write the SUI API according to this flowchart (I made that flowchart just for fun and to test the features of https://tldraw.com and it's actually the most complex flowchart I created till now…)

BTW The flowchart is read-only, you can't edit it…

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

SCRATCH---PIXEL

How do you store data? In MongoDB?
How much memory it takes you to store 3.7 million users IDs + usernames? (I don't think it's 40 MiB)

Before inserting user it checks whether there is already such user, I guess

Last edited by SCRATCH---PIXEL (July 17, 2023 17:45:50)

Sid72020123

SCRATCH---PIXEL wrote:
How do you store data? In MongoDB?
How much memory it takes you to store 3.7 million users IDs + usernames? (I don't think it's 40 MiB)

Before inserting user it checks whether there is already such user, I guess

Yes the data is stored in MongoDB…

Check the /get_data endpoint to know how the data is stored…

The total size of all the data (3.8M+ users) is roughly around 114.05 MB.

The previous versions of SUI used to check if the user exists but now it just upserts (updates + inserts) the user data which makes it more faster…

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

HPF2IH05

I did it and then bitdefender said it was dangerous so if it's okay can you take this down before others get scammed.

DifferentDance8

HPF2IH05 wrote:
I did it and then bitdefender said it was dangerous so if it's okay can you take this down before others get scammed.

Sid72020123 is a trustworthy figure in the AT scene (Advanced Topics, the subforum you just posted in). Why would he scam others?

Words? scoffs Who needs one? not when you have blinkies stamps - so confusing terminologies!

Sid72020123

HPF2IH05 wrote:
I did it and then bitdefender said it was dangerous so if it's okay can you take this down before others get scammed.

Wait. This cannot be true. It's maybe because of the “.space” domain or “.eu
.org” one. The API is completely safe and it is just used to get the data stored in the DB.

Last edited by Sid72020123 (July 19, 2023 10:21:04)

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

Sid72020123

DifferentDance8 wrote:
HPF2IH05 wrote:
I did it and then bitdefender said it was dangerous so if it's okay can you take this down before others get scammed.
Sid72020123 is a trustworthy figure in the AT scene (Advanced Topics, the subforum you just posted in). Why would he scam others?

Oh. Thank you so much

This day is one of the happiest days of my life since I joined Scratch.I am happy that someone trusts me.

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

mybearworld

HPF2IH05 wrote:
(#230)
I did it and then bitdefender said it was dangerous so if it's okay can you take this down before others get scammed.

Website scanners have a lot of false positives because they have to in order to catch most bad sites. SUI isn't dangerous.

Sid72020123

Sid72020123 wrote:
(#227)
I am going to re-write the SUI API according to this flowchart (I made that flowchart just for fun and to test the features of https://tldraw.com and it's actually the most complex flowchart I created till now…)

BTW The flowchart is read-only, you can't edit it…

I will start working on this soon when I get time…

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

Sid72020123

I am going to rewrite the API code (it may take over a month) and add new features to it. Also due to upcoming updates to Scratch and the API, I may completely pause the indexing program from 10th of August and resume it directly after 27th of August. During this period, you can still access and request the API to get the data…

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

josueart

Um, what about brute-forcing usernames? Nvm, there are like 40.481.239.792.225.648.784.462.962.744.968 possibilities.

Sadly, there are accounts with no connections, no followers, no following anyone, no projects shared, not in any studio, etc. How do you plan to index them?

My response may be inaccurate (i'm a human, humans commit miskates), so, please, remind that i have feelings and respond civilly.

“is that a zootopia reference?!!?!?!”

Sid72020123

josueart wrote:
(#237)
Um, what about brute-forcing usernames? Nvm, there are like 40.481.239.792.225.648.784.462.962.744.968 possibilities.

Sadly, there are accounts with no connections, no followers, no following anyone, no projects shared, not in any studio, etc. How do you plan to index them?

IDK. Anyways I've got so busy these days that I didn't even saw the status of the API.

And I forgot to announce that I stopped the main indexing program a few days ago because I didn't maintained it since months and it was bugging out…

I plan to re-write the API (with the development on Sunday every week for only 1 hour)

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

josueart

You should open-source the rewritten API so others can contribute.

My response may be inaccurate (i'm a human, humans commit miskates), so, please, remind that i have feelings and respond civilly.

“is that a zootopia reference?!!?!?!”

Sid72020123

josueart wrote:
You should open-source the rewritten API so others can contribute.

I have not started working on that yet…

Get the data of a random user in a list of 3.7M+ indexed Scratch users by visiting this URL! You can see the API docs here

Discuss Scratch