Exceeding the 50MB limit.

CatsUnited

I've noticed that this project made by comp09 exceeds the 50MB limit. WARNING: It's over 200MB!

Is it possible to do such a thing that you could break the 50MB limit? Okay, this may not be a very good idea to talk about though.

bottom text

comp09

It is possible to exceed the 50 MB limit if the assets are NOT added by the Scratch editor. Once the assets are packed into a sb2 file, for example using Kurt, you are free to upload it and add scripts regardless of its size.

Also, please do not attempt to open the costumes tab in that project CatsUnited linked above. It will almost certainly freeze and/or crash your browser.

Last edited by comp09 (March 18, 2015 00:59:59)

Visit the website of Andrew Sun!

MegaApuTurkUltra

comp09 wrote:
It is possible to exceed the 50 MB limit if the assets are NOT added by the Scratch editor. Once the assets are packed into a sb2 file, for example using Kurt, you are free to upload it and add scripts regardless of its size.

Also, please do not attempt to open the costumes tab in that project CatsUnited linked above. It will almost certainly freeze and/or crash your browser.

I'd thought about hacking assets onto the server myself, and then including them in the JSON, but I thought the server validates the JSON to make sure the contained assets don't exceed anything.

Apparently it works though. Cool!

This DOESN'T mean everyone should go make huge projects though. The limits are there for a reason. (I would include that 50MB is already making project load times slightly excessive, but 200MB would take forever :P - we don't all have google fiber you know)

$(".box-head")[0].textContent = "committing AT crimes since $whenever"

comp09

MegaApuTurkUltra wrote:
comp09 wrote:
It is possible to exceed the 50 MB limit if the assets are NOT added by the Scratch editor. Once the assets are packed into a sb2 file, for example using Kurt, you are free to upload it and add scripts regardless of its size.

Also, please do not attempt to open the costumes tab in that project CatsUnited linked above. It will almost certainly freeze and/or crash your browser.
I'd thought about hacking assets onto the server myself, and then including them in the JSON, but I thought the server validates the JSON to make sure the contained assets don't exceed anything.

Apparently it works though. Cool!

This DOESN'T mean everyone should go make huge projects though. The limits are there for a reason. (I would include that 50MB is already making project load times slightly excessive, but 200MB would take forever :P - we don't all have google fiber you know)

By the new FCC broadband definition (25 megabits/second), 200MB should only take 64 seconds to load.

Although Google Fiber would be nice to have over the Verizon FiPOOP I have, which is a “measly” 50 megabits both ways.
Thankfully, I don't have Comcast Poopfinity.

Visit the website of Andrew Sun!

AlanSlate

comp09 wrote:
It is possible to exceed the 50 MB limit if the assets are NOT added by the Scratch editor. Once the assets are packed into a sb2 file, for example using Kurt, you are free to upload it and add scripts regardless of its size.

Also, please do not attempt to open the costumes tab in that project CatsUnited linked above. It will almost certainly freeze and/or crash your browser.

can you send me a details tutorial of this ?
because I not quite understand⊙▽⊙

Last edited by AlanSlate (April 12, 2018 15:46:09)

ihgfedcba

does this still work in Scratch 3.0?

herohamp

ihgfedcba wrote:
does this still work in Scratch 3.0?

Please avoid necroposting

My website hamptonmoore.com
UMass Amherst All Campus Makerspace
Supporting the wife's SEO
Click for a Scratch based programming language I made

_nix

ihgfedcba wrote:
does this still work in Scratch 3.0?

Hypothetically, Scratch shouldn't have any problems opening >50MB projects – although, just like in 2.0, you probably won't be able to save or upload the project through the editor. You'd need to upload the assets by hand.

That said, although I don't think anyone's tested it recently, Scratch has serious trouble with memory usage when you open very large projects. See discussion here: https://github.com/LLK/scratch-gui/issues/1174

══ trans autistic lesbian enbydoggirls // 16 17 18 19 20, she/they ════
sparrows one word to the paragraph // <3 // ~(quasar) nebula

comp09

_nix wrote:
ihgfedcba wrote:
does this still work in Scratch 3.0?
Hypothetically, Scratch shouldn't have any problems opening >50MB projects – although, just like in 2.0, you probably won't be able to save or upload the project through the editor. You'd need to upload the assets by hand.

That said, although I don't think anyone's tested it recently, Scratch has serious trouble with memory usage when you open very large projects. See discussion here: https://github.com/LLK/scratch-gui/issues/1174

Case in point: one of my 200 MB projects required ~2 GB of RAM in Scratch 2. Now it requires (not a typo) more than 128 GB of RAM.

Visit the website of Andrew Sun!

_nix

comp09 wrote:
_nix wrote:
That said, although I don't think anyone's tested it recently, Scratch has serious trouble with memory usage when you open very large projects. See discussion here: https://github.com/LLK/scratch-gui/issues/1174
Case in point: one of my 200 MB projects required ~2 GB of RAM in Scratch 2. Now it requires (not a typo) more than 128 GB of RAM.

Is this still the case (i.e. as of today)? I know they've been working on memory usage improvements a fair deal since release (and even a couple are so recent they haven't been merged into scratch-gui yet, iirc).

Last edited by _nix (March 31, 2019 13:55:47)

══ trans autistic lesbian enbydoggirls // 16 17 18 19 20, she/they ════
sparrows one word to the paragraph // <3 // ~(quasar) nebula

CatsUnited

thanks for resurrecting this topic i always wanted to see 12 year old me make a topic

bottom text

Wettining

Can we talk about Scratch's memory problem? Not just in 3.0 but in 2.0 as well. Heck, Scratch 1.4 only used ~20 mb of RAM maybe 40 mb with a regular sized project. What's with Scratch and being a memory hog now? It's not as if it's the extra features since Scratch 1.4 mods made it possible to do everything Scratch 3.0 does and a lot more.
Kind of wish the Scratch team worked harder to make Scratch in something like C/C++ so that it could be compiled for both Desktop and emscriptened to the web. At least then we could get more control over the memory usage

Done with Scratch but want a little more? Try GP…
View all my posts:
1GErVELk2hM2WBK3TawRLR7AkV7SxyBSK6

dude341

comp09 wrote:
Case in point: one of my 200 MB projects required ~2 GB of RAM in Scratch 2. Now it requires (not a typo) more than 128 GB of RAM.

sorry excuse me

are you sure that's not a typo

(Obviously a memory leaking bug though, I can't see what else would cause that. If it's not a memory leak, well it just shows how much 3.0 is unfinished.)

Placeholder

MegaApuTurkUltra

Wettining wrote:
Scratch in something like C/C++

rust rust rust rust rust rust

$(".box-head")[0].textContent = "committing AT crimes since $whenever"

Wettining

MegaApuTurkUltra wrote:
Wettining wrote:
Scratch in something like C/C++
rust rust rust rust rust rust

I was thinking of including Rust but the code base is way too unstable atm plus those little security issues that popped up…
But yeah if Rust gets cleaned up a little :p

Done with Scratch but want a little more? Try GP…
View all my posts:
1GErVELk2hM2WBK3TawRLR7AkV7SxyBSK6

comp09

dude341 wrote:
comp09 wrote:
Case in point: one of my 200 MB projects required ~2 GB of RAM in Scratch 2. Now it requires (not a typo) more than 128 GB of RAM.
sorry excuse me

are you sure that's not a typo

(Obviously a memory leaking bug though, I can't see what else would cause that. If it's not a memory leak, well it just shows how much 3.0 is unfinished.)

That's not a typo. You can reduce it to ~80 GB if you don't have a graphics card though.

Last edited by comp09 (April 5, 2019 20:30:36)

Visit the website of Andrew Sun!

Jonathan50

comp09 wrote:
That's not a typo. You can reduce it to ~80 GB if you don't have a graphics card though.

Huh, wouldn't it need more, because the software renderer has to store everything in RAM?

Wettining wrote:
Kind of wish the Scratch team worked harder to make Scratch in something like C/C++ so that it could be compiled for both Desktop and emscriptened to the web. At least then we could get more control over the memory usage

I think that Scratch 2.0 shows that this is not necessary.

Not yet a Knight of the Mu Calculus.

MegaApuTurkUltra

Jonathan50 wrote:
comp09 wrote:
That's not a typo. You can reduce it to ~80 GB if you don't have a graphics card though.
Huh, wouldn't it need more, because the software renderer has to store everything in RAM?
Wettining wrote:
Kind of wish the Scratch team worked harder to make Scratch in something like C/C++ so that it could be compiled for both Desktop and emscriptened to the web. At least then we could get more control over the memory usage
I think that Scratch 2.0 shows that this is not necessary.

I still had issue with the 2.0 editor and large amounts of data (massive numbers of [small] sprites, giant lists, …), but the memory hogging was limited to the single GBs… IMO this is still unacceptable for the amount of data we are talking about. The problem is the entire project structure is presumably stored as ECMAScript objects, which are all hash tables. If written in a better language the data structures could be more space-efficient.

$(".box-head")[0].textContent = "committing AT crimes since $whenever"

comp09

MegaApuTurkUltra wrote:
Jonathan50 wrote:
comp09 wrote:
That's not a typo. You can reduce it to ~80 GB if you don't have a graphics card though.
Huh, wouldn't it need more, because the software renderer has to store everything in RAM?
Wettining wrote:
Kind of wish the Scratch team worked harder to make Scratch in something like C/C++ so that it could be compiled for both Desktop and emscriptened to the web. At least then we could get more control over the memory usage
I think that Scratch 2.0 shows that this is not necessary.
I still had issue with the 2.0 editor and large amounts of data (massive numbers of [small] sprites, giant lists, …), but the memory hogging was limited to the single GBs… IMO this is still unacceptable for the amount of data we are talking about. The problem is the entire project structure is presumably stored as ECMAScript objects, which are all hash tables. If written in a better language the data structures could be more space-efficient.

Based on the behavior I have observed, here is an explanation for the memory usage: Scratch 3.0 creates WebGL textures for all costumes in a project when it is loaded. This involves rasterizing SVG costumes too. These textures require about 32 bits per pixel of texture data (since it's RGBA).

If you have a discrete graphics card (haven't tested with integrated graphics, those can be a bit weird), it uploads the texture to the GPU's memory while also keeping a copy in system memory, because why not (e.g. if you've ever experienced a graphics driver crash, your browser doesn't suddenly lose all its tabs and merely falls back to software rendering). Projects with a lot of costumes will inevitably use more graphics memory than the 1-8 GB available in consumer graphics cards, so the graphics driver will “swap” the textures to system memory. So now we've got two copies of the same data in RAM.

Sidenote: The reason why textures are “uploaded” like this is because GPUs are connected to the CPU and system memory via a “slow” PCIe connection, which maxes out at 128 GBit/s on a typical desktop and may be as low as 32 Gbit/s. The computer I'm typing this post on has a GPU with 3272 GBit/s of graphics memory bandwidth.

If you don't have a graphics card, Firefox will emulate WebGL with a software renderer. This seems to have some optimizations, where “upload texture to GPU” is really just copying a pointer. So RAM usage is about half of what it would be if you had a graphics card.

Then, to account for the 128+ GB of usage, we multiply the theoretical memory usage by a 2-4x fudge factor to account for various inefficiencies (scaling, collision detection, alpha masks, etc.) in Scratch's code. Memory usage patterns have now been explained.

Of course, this is vaguely informed speculation based on cursory glances at scratch-render code and observing what happens when I torture stress-test my laptop…

p.s. ECMAscript objects aren't really memory inefficient. They're actually extremely well-optimized in modern browsers, because they're used everywhere. They only become memory-inefficient when you do something stupid that causes the engine to bail out and use a hashtable instead of hidden classes or whatever voodoo V8 is using nowadays. However, sometimes it might be worthwhile to use features like typed arrays to squeeze every last bit of memory efficiency out of your code.

Last edited by comp09 (April 8, 2019 06:18:06)

Visit the website of Andrew Sun!

MegaApuTurkUltra

comp09 wrote:
p.s. ECMAscript objects aren't really memory inefficient. They're actually extremely well-optimized in modern browsers, because they're used everywhere. They only become memory-inefficient when you do something stupid that causes the engine to bail out and use a hashtable instead of hidden classes or whatever voodoo V8 is using nowadays. However, sometimes it might be worthwhile to use features like typed arrays to squeeze every last bit of memory efficiency out of your code.

or use rust

$(".box-head")[0].textContent = "committing AT crimes since $whenever"

Discuss Scratch