Every project with an ID (the number in a project link) higher than 643014898 seems to not have the ability to get on trending or to show up when you search for them.

To verify this, you can take any project with an ID above 643014898 and try to search for it with the search bar and I can guarantee you that it will not show up, which means that it also do not have the ability to get on the trending page (for a project to appear on trending, it needs to show up when you search for it). So, if you share a project right now with an ID above 643014898 it will not have any chance to get on trending no matter how well it does. Now, if you create a new project, the ID will be ~700 000 000, so millions of projects have not been indexed.

So, every newly created projects are not indexed anymore for some reason. It was @colinmacc who actually found that the highest ID number that was indexed was 643014898. Both me and him also did a test where we shared 2 projects at the same time (one with a high ID and one with a low ID) and, for both of us, only the project with the low ID showed up with the search bar.

colinmacc's two test projects:
https://scratch.mit.edu/projects/309448663/
https://scratch.mit.edu/projects/681414866/

Even if you go in the recent tab (which should normaly be filled with high ID projects) in the explore page, the projects all have an ID below 643014898.

Now, the only way for your project to have the ability to get on trending is to save the project, load it into an old project with an ID < 643014898 and to share this project instead. This method is not possible for scratchers who joined in the last months who don't have a project with an ID < 643014898 in their folder.



This problem is significant for many reasons:

- The search bar is useless for new projects (exemple: The search bar used to be very usefull for teachers who wanted to share a project with their student but this way to do it is not longer possible)

- The trending page is now completly meaningless (What is a trending page if it can only show the projects with an ID < 643014898 which represent probably 0.1% of the recently shared projects? Thousands of exceptional projects do not get the recognition they deserve only because their ID make it impossible for them to show up)

-This problem is going to eventually get even worse (With time, a lot of people are going to realize the issue and are going start avoiding it by loading the project they want to share in an old project each time they share a new one. But, what is going to happen when people won't have any old projects left? I don't even know what would happen to the explore page if this ever happens.

I made this post mainly to point out the bug to the Scratch Team but also so that as many people as possible learn about it and use the method I presented to make sure their projects can show up.


Thank you! :-)

that is so weird. sorry I have no idea how to solve it, I guess the scratch team will be the only one able to.

It's very hard to argue with the statistics on this one. Something fishy about this, for sure.

That's so weird… 643014898 seems really arbitrary, so I'm going to throw a guess in what the problem is based on nothing but this rather random number!


Wild speculation to follow!

It feels like there's a database inner join going on and something has changed on the population of one of the tables turning the join effectively into a where (or filter) reducing the dataset to everything that worked before that change.

So and example scenario would be if there was a projectTypes table (which contained a look up ID and Name for each project type) and when a project was stored they set the projectType…

However, overtime they realised projectTypes weren't used so a change was made and project type was no longer saved with a project (just leaving null values in that field). The search query used and the join said Select * from projects p inner join projectTypes on p.pTypeID = pt.pTypeID … if the projects table stopped populating pTypeID this would effectively turn into a filter rather than a join. (The solution to this would be to change the inner to an Outer join and all would be good)

It's as good a theory as any at this point!

seems a lil sus to me…

RobFarley74 wrote:

That's so weird… 643014898 seems really arbitrary, so I'm going to throw a guess in what the problem is based on nothing but this rather random number!


Wild speculation to follow!

It feels like there's a database inner join going on and something has changed on the population of one of the tables turning the join effectively into a where (or filter) reducing the dataset to everything that worked before that change.

So and example scenario would be if there was a projectTypes table (which contained a look up ID and Name for each project type) and when a project was stored they set the projectType…

However, overtime they realised projectTypes weren't used so a change was made and project type was no longer saved with a project (just leaving null values in that field). The search query used and the join said Select * from projects p inner join projectTypes on p.pTypeID = pt.pTypeID … if the projects table stopped populating pTypeID this would effectively turn into a filter rather than a join. (The solution to this would be to change the inner to an Outer join and all would be good)

might just be the search filter not being able to handle such a huge number

This is a pretty serious issue, I hope the scratch team sees this

If im not wrong, you can still see these projects if you put your searach results into “search for popular” and not “search for trending”, I will guess this number is random because that was the last project created before some kind of update on scratch caused no more newly created projects to be able to show as trending on the search page

Another theory that just came to me. Maybe it’s something temporary someone put in to clear a backlog, and they forgot to take it out again?

i think this is just the regular occurrence of the explore indexer thing taking time to index all the new projects.

i don't know how accurate it is anymore, but there's a pretty detailed explanation of search/the trending algorithm at https://dspace.mit.edu/bitstream/handle/1721.1/106393/967666601-MIT.pdf. essentially it lives as a separate system that has access to the database, but has to index projects independently for them to show up on search.

perhaps it's automatically updating the index if project details change, but not if new projects are made?

there could be a massive backlog right now that's preventing it from showing recent projects.

Jeffalo wrote:

i think this is just the regular occurrence of the explore indexer thing taking time to index all the new projects.

i don't know how accurate it is anymore, but there's a pretty detailed explanation of search/the trending algorithm at https://dspace.mit.edu/bitstream/handle/1721.1/106393/967666601-MIT.pdf. essentially it lives as a separate system that has access to the database, but has to index projects independently for them to show up on search.

perhaps it's automatically updating the index if project details change, but not if new projects are made?

there could be a massive backlog right now that's preventing it from showing recent projects.

Well all I know is after weeks of not getting projects indexed, I've started recycling old project id's of deleted projects that have never been shared, and the projects are getting indexed within 24 hours.

So if the scratch team wants to leave things as they are that's fine by me - I have hundreds of deleted projects.

Other people may think otherwise!

colinmacc wrote:

Well all I know is after weeks of not getting projects indexed, I've started recycling old project id's of deleted projects that have never been shared, and the projects are getting indexed within 24 hours.

So if the scratch team wants to leave things as they are that's fine by me - I have hundreds of deleted projects.

Other people may think otherwise!

I will also use an old project everytime I share a new one. I have enough empty projects for the rest of the year lol. I never thought these projects would ever be usefull! I still think the ST should fix this soon though, because most people still don't know this method and are still sharing theirs projects hoping in vain that it might get on trending.

I messaged the ST about a privacy concern related to this bug. This is what they said:

Notice the date is April 8, which was almost a month ago. Should it be taking this long to index all the recent projects? Anyways, they say that they can't do anything about it, we just need to wait.

This is wat turbowarp said:
May 7, 2022 - Important notice regarding unshared projects

The latest code changes to Scratch indicate the Scratch Team is finally making unshared projects inaccessible to anyone who isn't the owner of the project.

We aren't sure when the changes will happen or their exact impact, but we expect that unshared projects will no longer be viewable or embeddable in TurboWarp or other third-party websites. This includes your own unshared projects.

Kitten-da-Cat wrote:

This is wat turbowarp said:
May 7, 2022 - Important notice regarding unshared projects

The latest code changes to Scratch indicate the Scratch Team is finally making unshared projects inaccessible to anyone who isn't the owner of the project.

We aren't sure when the changes will happen or their exact impact, but we expect that unshared projects will no longer be viewable or embeddable in TurboWarp or other third-party websites. This includes your own unshared projects.

Wow thanks for the info! This might actually be the cause of the problem! We'll see!

alexandretherrien wrote:

Kitten-da-Cat wrote:

This is wat turbowarp said:
May 7, 2022 - Important notice regarding unshared projects

The latest code changes to Scratch indicate the Scratch Team is finally making unshared projects inaccessible to anyone who isn't the owner of the project.

We aren't sure when the changes will happen or their exact impact, but we expect that unshared projects will no longer be viewable or embeddable in TurboWarp or other third-party websites. This includes your own unshared projects.

Wow thanks for the info! This might actually be the cause of the problem! We'll see!

i don't think it is. that's a completely separate change that hasn't been implemented yet

alexandretherrien wrote:

Kitten-da-Cat wrote:

This is wat turbowarp said:
May 7, 2022 - Important notice regarding unshared projects

The latest code changes to Scratch indicate the Scratch Team is finally making unshared projects inaccessible to anyone who isn't the owner of the project.

We aren't sure when the changes will happen or their exact impact, but we expect that unshared projects will no longer be viewable or embeddable in TurboWarp or other third-party websites. This includes your own unshared projects.

Wow thanks for the info! This might actually be the cause of the problem! We'll see!

I don't quite see how this is connected to the search (indexer) issue? But the ST comment about the delay is interesting. We have seen that in the past, but back then it took days or weeks to catch up, not months; and AFAIR only for some projects, not all.

This is really strange, and could get worse. I don't know how to fix this. The ST are the only ones that will be able to fix the problem. I have noticed the explore page not really changing recently, but I never thought it was a bug.

filmlover12 wrote:

This is really strange, and could get worse. I don't know how to fix this. The ST are the only ones that will be able to fix the problem. I have noticed the explore page not really changing recently, but I never thought it was a bug.

RL1123 wrote:

I messaged the ST about a privacy concern related to this bug. This is what they said:

Notice the date is April 8, which was almost a month ago. Should it be taking this long to index all the recent projects? Anyways, they say that they can't do anything about it, we just need to wait.

The ST says that they aren't able to fix this.

ArnoHu wrote:

I don't quite see how this is connected to the search (indexer) issue? But the ST comment about the delay is interesting. We have seen that in the past, but back then it took days or weeks to catch up, not months; and AFAIR only for some projects, not all.

Hmm, I might be able to explain why this time it takes much longer than in past incidents. Looking at the current project ID, there have been approximately 44,000,000 projects created since project ID 643014898 and a smaller amount has been shared. I estimate that to be around 5 million, looking at the Statistics page and summing the projects since February. That's still an insane amount of projects and I don't think Scratch has ever had this level of activity before. But still, it shouldn't have this long of a wait…