A while ago, I read that projects with URLs that start with 97 or higher weren't able to get indexed and on trending

yadayadayadagoodbye wrote:

Because every set period of time, it has to restart from index 0, thus, as the amount of projects increase, projects further down the list take longer to get indexed (as you'd have to get lucky for it to get to you in time). Apparently the ST has stopped the indexer from indexing really early projects, which fixed it for a while, but due to the rapid growth in amount of projects, it is again failing to get past the 960 million project range (from what i've heard).

And don't ask for a source, I have no source, I got this from some random (trustworthy) forum post but I cannot link to it as I have no idea where or what its called.

And with the billionth project being made today, any created project now would default to having a URL that starts with 10, so does this push away the previous URL/indexing problem and makes getting on trending possible with projects under a billionth ID? Or is there still some catch to all of that with the indexer acting funky?

LazyEyeTV wrote:

And with the billionth project being made today, any created project now would default to having a URL that starts with 10, so does this push away the previous URL/indexing problem and makes getting on trending possible with projects under a billionth ID? Or is there still some catch to all of that with the indexer acting funky?

It counts up from zero, what the url starts with des not matter.

ideapad-320 wrote:

LazyEyeTV wrote:

And with the billionth project being made today, any created project now would default to having a URL that starts with 10, so does this push away the previous URL/indexing problem and makes getting on trending possible with projects under a billionth ID? Or is there still some catch to all of that with the indexer acting funky?

It counts up from zero, what the url starts with des not matter.

How long does it take? The indexer started from zero again two weeks ago.

ideapad-320 wrote:

LazyEyeTV wrote:

And with the billionth project being made today, any created project now would default to having a URL that starts with 10, so does this push away the previous URL/indexing problem and makes getting on trending possible with projects under a billionth ID? Or is there still some catch to all of that with the indexer acting funky?

It counts up from zero, what the url starts with des not matter.

How long does it take? The indexer started from zero again two weeks ago.

Yes it can, actually, it is on trending now.

Projects with an id above 970 000 000 will never get indexed until:

1. Something is done by the ST to make the indexer be able to keep up with the rate of new projects being shared.

2. There is a massive decrease in new projects causes, for example, by holidays.

3. Enough time has passed. (Probably months)

Because the backlog of unindexed projects in increasing every day, if nothing is done, the problem will only get worse and it might reach a point where high id projects are simply never indexed.


By the way the 970 000 000 limit changes every day. Some day the indexer might be able to reach 965 000 000, but the next day, due to more new projects, it might only be able to reach id 920 000 000, thus every project with an id above this number not getting indexed this day. In the future some projects with an ID above 970 000 000 will be indexed, but, by the time this is the case, new projects will have an id of like 1 100 000 000 for example. This is all because the indexer can’t keep up.

alexandretherrien wrote:

Project with an id above 970 000 000 will never get indexed until:

1. Something is done by the ST to make the indexer be able to keep up with the rate of new projects being shared.

2. There is a massive decrease in new projects causes, for example, by holidays.

3. Enough time has passed. (Probably months)

Because the backlog of unindexed projects in increasing every day, if nothing is done, the problem will only get worse and it might reach a point where high id projects are simply never indexed.


By the way the 970 000 000 limit changes every day. Some day the indexer might be able to reach 965 000 000, but the next day, due to more new projects, it might only be able to reach id 920 000 000, thus every project with an id above this number not getting indexed this day. In the future some projects with an ID above 970 000 000 will be indexed, but, by the time this is the case, new projects will have an id of like 1 100 000 000 for example. This is all because the indexer can’t keep up.

When did this lag start? I've noticed it myself for the first time today, because I shared a project and it didn't get to trending. But I see some people talking about it a month ago. You seem fairly knowledgable.

Chrome_Cat wrote:

alexandretherrien wrote:

Project with an id above 970 000 000 will never get indexed until:

1. Something is done by the ST to make the indexer be able to keep up with the rate of new projects being shared.

2. There is a massive decrease in new projects causes, for example, by holidays.

3. Enough time has passed. (Probably months)

Because the backlog of unindexed projects in increasing every day, if nothing is done, the problem will only get worse and it might reach a point where high id projects are simply never indexed.


By the way the 970 000 000 limit changes every day. Some day the indexer might be able to reach 965 000 000, but the next day, due to more new projects, it might only be able to reach id 920 000 000, thus every project with an id above this number not getting indexed this day. In the future some projects with an ID above 970 000 000 will be indexed, but, by the time this is the case, new projects will have an id of like 1 100 000 000 for example. This is all because the indexer can’t keep up.

When did this lag start? I've noticed it myself for the first time today, because I shared a project and it didn't get to trending. But I see some people talking about it a month ago. You seem fairly knowledgable.

This bug is quite old (here's the same bug in 2022). The problem is caused by the fact that the crawler (indexer) has to cycle from project ID 0 to the highest ID everyday. The problem is that when the number of new shared projects is higher than what the indexer can index in a day, it can't reach the high ids before having to restart its cycle from ID 0 which causes an increasingly big backlog of unindexed shared projects. As far as I'm aware, it's the third time that this issue has appeared. The last time, was about 6 months ago. Back then, I reported the issue to a Scratch dev which took the issue seriously. A few days later the Scratch Team made it so the indexer wouldn't start its cycle at ID 0. This fixed the issue (at least for high id projects), but then it looks like the changes were reverted because, in february, the bug came back.

alexandretherrien wrote:

This bug is quite old (here's the same bug in 2022). The problem is caused by the fact that the indexer has to cycle from project ID 0 to the highest ID everyday. The problem is that when the number of new shared projects is higher than what the indexer can index in a day, it can't reach the high ids before having to restart its cycle from ID 0 which causes an increasingly big backlog of unindexed shared projects. As far as I'm aware, it's the third time that this issue has appeared. The last time, was about 6 months ago. Back then, I reported the issue to a Scratch dev which took the issue seriously. A few days later the Scratch Team made it so the indexer wouldn't start its cycle at ID 0. This fixed the issue (at least for high id projects), but then it looks like the changes were reverted because, in february, the bug came back.

Wouldn't they cycle through ids in a more complex manner? Like high priority to those on top of trending and recently shared projects?

Its gonna be hard to get on trending, especially for newer scratchers who have only a few or no projects less than a billion

you can just transp[ort a new proejct into an old file

HardCharger4 wrote:

you can just transp[ort a new proejct into an old file

that wouldn't work because new scratchers (ones under 9 month time) dont really have old projects that the index can read