alexandretherrien wrote:

Great! But i'm pretty sure the indexer did not get fixed at all (I explain why on the #108 post) If some more projects are getting indexed right now it's probably because the number of project posted everyday is getting lower. At least that's my theory. :b

Buuuut heeeeyy that's just a theory a gaaaame theory thanks for watching
I don't see how less projects being shared means that the indexer suddenly works

musicROCKS013 wrote:

alexandretherrien wrote:

Great! But i'm pretty sure the indexer did not get fixed at all (I explain why on the #108 post) If some more projects are getting indexed right now it's probably because the number of project posted everyday is getting lower. At least that's my theory. :b

Buuuut heeeeyy that's just a theory a gaaaame theory thanks for watching
I don't see how less projects being shared means that the indexer suddenly works

The fact that the indexer index a bunch of projects with a certain ID a day and then a bunch of projects with way lower IDs the next day (like a few days ago it was indexing IDs of 650 000 000 and the next day project with IDs lower than 640 000 000) makes me think that the indexer might not index every project one after the other in order, but maybe starts at the lowest number and works its way up everyday.

colinmacc wrote:

TimMcCool wrote:

Scratch's indexer has been broken for a long time. I've noticed that projects with a low project id are usually indexed much faster than projects with a high level id and therefore more likely to appear on the explore page. Somtimes, the trending algorithm will only index projects if their project id is lower than a specific number. It has been like that since early 2021 (since the reshare glitch was patched).

I guess it's possible it starts at the lowest number and works its way up, and there have been so many shared projects recently that it never gets to the end of the list before restarting the next day.

Maybe we've got this completely wrong

alexandretherrien wrote:

-mega snip-

Ahhh… I see. That makes more sense. Like, the projects shared at the beginning of the day are more likely to index than projects shared at the end of the day?

musicROCKS013 wrote:

alexandretherrien wrote:

-mega snip-

Ahhh… I see. That makes more sense. Like, the projects shared at the beginning of the day are more likely to index than projects shared at the end of the day?

No, I don't think so. The indexer would start indexing the project 000 000 001 then 000 000 002 then, 2, 3, 4… and by the end of the day, it would reach something like 650 000 000 and then go back to 0. So projects after 650 000 000 would not be indexed

alexandretherrien wrote:

No, I don't think so. The indexer would start indexing the project 000 000 001 then 000 000 002 then, 2, 3, 4… and by the end of the day, it would reach something like 650 000 000 and then go back to 0. So projects after 650 000 000 would not be indexed

Oh. It indexes them again and again every day? That's interesting…

musicROCKS013 wrote:

alexandretherrien wrote:

No, I don't think so. The indexer would start indexing the project 000 000 001 then 000 000 002 then, 2, 3, 4… and by the end of the day, it would reach something like 650 000 000 and then go back to 0. So projects after 650 000 000 would not be indexed

Oh. It indexes them again and again every day? That's interesting…

No, but imagine every time someone shares a project or changes its title or description, etc, a piece of paper with that project id gets put in a box. Then at the end of the day the box is emptied one at a time, but rather than in the order they were added, in sequential project id order. Except before the box is fully emptied it stops, and the next day begins.

On days with fewer changes, the indexer gets further along the list. But crucially it never seems to get to the end. But it probably catches up a bit over the weekends, and it should catch up a lot over the summer.

(I mention the title and description because when I first shared Wordle in January it was called “Wordle for Scratch”, and that is the description that appeared in search. About a week later I changed the description to just “Wordle”, but that change didn't appear in the search results until last week. The project id is ~630million I think)

colinmacc wrote:

musicROCKS013 wrote:

alexandretherrien wrote:

No, I don't think so. The indexer would start indexing the project 000 000 001 then 000 000 002 then, 2, 3, 4… and by the end of the day, it would reach something like 650 000 000 and then go back to 0. So projects after 650 000 000 would not be indexed

Oh. It indexes them again and again every day? That's interesting…

No, but imagine every time someone shares a project or changes its title or description, etc, a piece of paper with that project id gets put in a box. Then at the end of the day the box is emptied one at a time, but rather than in the order they were added, in sequential project id order. Except before the box is fully emptied it stops, and the next day begins.

On days with fewer changes, the indexer gets further along the list. But crucially it never seems to get to the end. But it probably catches up a bit over the weekends, and it should catch up a lot over the summer.

(I mention the title and description because when I first shared Wordle in January it was called “Wordle for Scratch”, and that is the description that appeared in search. About a week later I changed the description to just “Wordle”, but that change didn't appear in the search results until last week. The project id is ~630million I think)

thats a terrible method, because it will never be properly caught up

i made an indexing test https://scratch.mit.edu/projects/698557930/ i will make another one at the end of the day

saverofthewo wrote:

colinmacc wrote:

musicROCKS013 wrote:

alexandretherrien wrote:

No, I don't think so. The indexer would start indexing the project 000 000 001 then 000 000 002 then, 2, 3, 4… and by the end of the day, it would reach something like 650 000 000 and then go back to 0. So projects after 650 000 000 would not be indexed

Oh. It indexes them again and again every day? That's interesting…

No, but imagine every time someone shares a project or changes its title or description, etc, a piece of paper with that project id gets put in a box. Then at the end of the day the box is emptied one at a time, but rather than in the order they were added, in sequential project id order. Except before the box is fully emptied it stops, and the next day begins.

On days with fewer changes, the indexer gets further along the list. But crucially it never seems to get to the end. But it probably catches up a bit over the weekends, and it should catch up a lot over the summer.

(I mention the title and description because when I first shared Wordle in January it was called “Wordle for Scratch”, and that is the description that appeared in search. About a week later I changed the description to just “Wordle”, but that change didn't appear in the search results until last week. The project id is ~630million I think)

thats a terrible method, because it will never be properly caught up

I didn't invent it.. Also I'm only speculating based on observed evidence. I could be wrong. I usually am…

colinmacc wrote:

saverofthewo wrote:

colinmacc wrote:

musicROCKS013 wrote:

alexandretherrien wrote:

No, I don't think so. The indexer would start indexing the project 000 000 001 then 000 000 002 then, 2, 3, 4… and by the end of the day, it would reach something like 650 000 000 and then go back to 0. So projects after 650 000 000 would not be indexed

Oh. It indexes them again and again every day? That's interesting…

No, but imagine every time someone shares a project or changes its title or description, etc, a piece of paper with that project id gets put in a box. Then at the end of the day the box is emptied one at a time, but rather than in the order they were added, in sequential project id order. Except before the box is fully emptied it stops, and the next day begins.

On days with fewer changes, the indexer gets further along the list. But crucially it never seems to get to the end. But it probably catches up a bit over the weekends, and it should catch up a lot over the summer.

(I mention the title and description because when I first shared Wordle in January it was called “Wordle for Scratch”, and that is the description that appeared in search. About a week later I changed the description to just “Wordle”, but that change didn't appear in the search results until last week. The project id is ~630million I think)

thats a terrible method, because it will never be properly caught up

I didn't invent it.. Also I'm only speculating based on observed evidence. I could be wrong. I usually am…

actually you are probably right

Thx for letting us know!

alexandretherrien wrote:

Every project with an ID (the number in a project link) higher than 643014898 seems to not have the ability to get on trending or to show up when you search for them.

To verify this, you can take any project with an ID above 643014898 and try to search for it with the search bar and I can guarantee you that it will not show up, which means that it also do not have the ability to get on the trending page (for a project to appear on trending, it needs to show up when you search for it). So, if you share a project right now with an ID above 643014898 it will not have any chance to get on trending no matter how well it does. Now, if you create a new project, the ID will be ~700 000 000, so millions of projects have not been indexed.

So, every newly created projects are not indexed anymore for some reason. It was @colinmacc who actually found that the highest ID number that was indexed was 643014898. Both me and him also did a test where we shared 2 projects at the same time (one with a high ID and one with a low ID) and, for both of us, only the project with the low ID showed up with the search bar.

colinmacc's two test projects:
https://scratch.mit.edu/projects/309448663/
https://scratch.mit.edu/projects/681414866/

Even if you go in the recent tab (which should normaly be filled with high ID projects) in the explore page, the projects all have an ID below 643014898.

Now, the only way for your project to have the ability to get on trending is to save the project, load it into an old project with an ID < 643014898 and to share this project instead. This method is not possible for scratchers who joined in the last months who don't have a project with an ID < 643014898 in their folder.



This problem is significant for many reasons:

- The search bar is useless for new projects (exemple: The search bar used to be very usefull for teachers who wanted to share a project with their student but this way to do it is not longer possible)

- The trending page is now completly meaningless (What is a trending page if it can only show the projects with an ID < 643014898 which represent probably 0.1% of the recently shared projects? Thousands of exceptional projects do not get the recognition they deserve only because their ID make it impossible for them to show up)

-This problem is going to eventually get even worse (With time, a lot of people are going to realize the issue and are going start avoiding it by loading the project they want to share in an old project each time they share a new one. But, what is going to happen when people won't have any old projects left? I don't even know what would happen to the explore page if this ever happens.

I made this post mainly to point out the bug to the Scratch Team but also so that as many people as possible learn about it and use the method I presented to make sure their projects can show up.


Thank you! :-)

So, does anybody actually know what is causing this?

musicROCKS013 wrote:

So, does anybody actually know what is causing this?

Recently we think it’s just volume of new and changed projects

colinmacc wrote:

musicROCKS013 wrote:

So, does anybody actually know what is causing this?

Recently we think it’s just volume of new and changed projects

i hope the indexer has more priority over new than changed, i would be fine if i went months without the indexer processing my new instructions

musicROCKS013 wrote:

So, does anybody actually know what is causing this?

The exact origin of the issue is not really known, but one thing's sure: Low ID projects index way faster and reliably than high ID projects.

So loading the project you want to share into an older project is still very usefull. (Plus, I heard that if you do that, you can avoid the bug that causes projects with more then 100 loves in the first day to not show)

saverofthewo wrote:

colinmacc wrote:

musicROCKS013 wrote:

So, does anybody actually know what is causing this?

Recently we think it’s just volume of new and changed projects

i hope the indexer has more priority over new than changed, i would be fine if i went months without the indexer processing my new instructions

I think instructions changes update immediately, but title updates take a while to show up to others.

Alicorn468 wrote:

Thx for letting us know!

alexandretherrien wrote:

*snip*

Thank you! :-)

Please don't quote the original post!

saverofthewo wrote:

colinmacc wrote:

musicROCKS013 wrote:

So, does anybody actually know what is causing this?

Recently we think it’s just volume of new and changed projects

i hope the indexer has more priority over new than changed, i would be fine if i went months without the indexer processing my new instructions

I agree.

I think it's just fine working because all of the projects on recent were published today. I might be wrong, though…

Draconite_Games wrote:

I think it's just fine working because all of the projects on recent were published today. I might be wrong, though…

Not really. It has always been like this even since the bug appeared. The problem is rather that they now all have an ID below a certain number.