Story time: I ran into a unknown problem when coding a world generator that caused layers of the world to be missing. The world required 262144 (64x64x64) voxels. After an hour or so frantically pulling my hair out, I then discovered that my world's voxel list wouldn't go past 200000 in length. That is frustratingly low for many of my complex projects, some which need at least twice that much storage.

You might say I should use more lists if I want more storage. Problem is, I already have multiple lists holding simple information, and 3.0 simply refuses to allow more lists to be created on demand. There's just no way to get around this unless I build some crazy switch for choosing which list to retrieve info from, which only raises the limit but doesn't remove it outright. Each time I would do this it would severely impact performance and make the project a huge waste of time, both for coding and waiting.

I understand that, at least for previous versions of Scratch, huge lists could crash the application, freeze the computer, and make saving the project difficult or impossible for a variety of reasons. This I know all too well, however, I may add that such a limit restricts the scope of projects by culling out the more advanced kinds like per-pixel image processing, world generation, and 3d engines with high-poly meshes. I highly suggest better provisions and optimizations to minimize the chance of problems occurring.

So a hard limit of 200000 items is not enough. Please, I'm asking for 1 million items, maybe just half that, if removing it isn't an option the ST is willing to consider.

Update: This comment by @thisandagain helps to shed some light:

thisandagain wrote:

The limit was really a performance and UX compromise. During development we did some stress testing and found that large lists were causing performance issues on many of our lower-end test devices so we started to enforce a limit to keep project performance acceptable. One of the main reasons we were concerned about this is that in 3.0 adding items to a list (forever -> add (foo) to [list]) is *much* faster (and thus easier to get into a bad state without a limit).

However, I do have a few criticisms about this.

First, the data analysis did not include runtime list sizes, only those stored alongside their projects, so the distribution was biased for static data. Many projects of mine, and perhaps of many others, rely on dynamic list allocation to populate their games and simulations with sufficient data, which is usually cleared before the project is saved to reduce bandwidth/project size. The only times it's not cleared are, for example, debugging and sharing a specific project state.

Second, adding items ad infinitum to a list ought to happen as expected. There is practically no reason to not give the benefit of the doubt to Scratchers on behalf of their ability to understand their code. Code that does exactly what you tell it to do, no side effects, is the best way to learn to respect it. If it does stuff without you knowing, like limiting what you create, you start to distrust your own code and yourself (you really shouldn't). It's needless to say, mistakes are part of learning; if you accidentally make an infinite loop of adding things to a list, just be careful next time, it's not going to destroy your computer but it would leave a good reminder to double check your flow control.

Third, while it is a valid concern that performance across devices be as smooth as possible, there are still several ways people can crash Scratch, even accidentally, just by how the VM works – using recursion and cloning. Like lists, these two significant features of Scratch are also very dangerous when used improperly. I would go as far to wager that they are more dangerous than unbounded lists because of their easy ability to lock the CPU in a high usage loop, meanwhile filling a list can take a while to get up to a few hundred MB and it won't even brick the CPU. Futhermore, most devices will resort to paging on disk when there is low available RAM, so in my opinion, the performance issue is only ever present for devices that aren't even meant to run Scratch projects of mid-to-high caliber.

Fourth, the stress-testing on low-end devices was expected to yield bad performance results, but that shouldn't mean all devices must suffer because of it. Instead, simply recommend hardware minimums so that educators know not to use computers from the early 2000's.

I would like to remind everyone, including the Scratch Team themselves, about the Scratch Design Goals, which state:

lightnin wrote:

If Scratch was a room, it would have a low floor, wide walls, and a high ceiling.

* Low floor: It should be easy to climb in and get started with Scratch - even for Scratchers who have no experience programming.

* Wide walls: Scratchers should be able to make all kinds of things with Scratch - not just animations and games, but news programs, science experiments - things we can't even imagine.

* High Ceiling: Even though it's easy for someone who is new to programming to get started with Scratch, it should still be possible to make complex stuff.

– Source

I'm entirely for upholding this brilliant philosophy by @lightnin, one of the co-founders of Scratch. His vision of Scratch, to be a tool for beginners and experts alike with near-limitless potential for creativity, is a beacon of guidance to us all. However, I have reason to believe the Scratch Team is receding on this ideal in favor of catering to their younger, less experienced audience, thus bringing the ceiling down with the floor. So as my final proposal, I would like the list limit of 200k items to only affect whether a project is saved on server or not, and use a higher hard limit for runtime performance.

No support, workaroundable.



Jokes aside, I greatly support this suggestion as I believe it was one of the biggest mistakes of scratch 3.0, not only breaking many projects from scratch 2.0 but also rendering Turing completeness an exploit of concurrency and recursion.
Although it is workaroundable as shown above that solution is O(n) for accessing and replacing content compared to the O(1) of lists, making it unfeasible in almost all cases.
(More information on the turing completeness of Scratch 3.0 here: https://scratch-mit-edu.ezproxyberklee.flo.org/discuss/topic/318309/)

Support. I feel like the Scratch Team is really hurting more advanced Scratchers with these recent changes. (Not just the list limit, but the cloud variable limit too!)

Zro716 wrote:

Story time: I ran into a unknown problem when coding a world generator that caused layers of the world to be missing. The world required 262144 (64x64x64) voxels. After an hour or so frantically pulling my hair out, I then discovered that my world's voxel list wouldn't go past 200000 in length. That is frustratingly low for many of my complex projects, some which need at least twice that much storage.

You might say I should use more lists if I want more storage. Problem is, I already have multiple lists holding simple information, and 3.0 simply refuses to allow more lists to be created on demand. There's just no way to get around this unless I build some crazy switch for choosing which list to retrieve info from, which only raises the limit but doesn't remove it outright. Each time I would do this it would severely impact performance and make the project a huge waste of time, both for coding and waiting.

I understand that, at least for previous versions of Scratch, huge lists could crash the application, freeze the computer, and make saving the project difficult or impossible for a variety of reasons. This I know all too well, however, I may add that such a limit restricts the scope of projects by culling out the more advanced kinds like per-pixel image processing, world generation, and 3d engines with high-poly meshes. I highly suggest better provisions and optimizations to minimize the chance of problems occurring.

So a hard limit of 200000 items is not enough. Please, I'm asking for 1 million items, maybe just half that, if removing it isn't an option the ST is willing to consider.

Perhaps instead of creating a “crazy switch” you could use clones holding local lists. This bumps it up to 60 million items (provided you aren't using clones for something else), while not infinite should be enough.

badatprogrammingibe wrote:

Perhaps instead of creating a “crazy switch” you could use clones holding local lists. This bumps it up to 60 million items (provided you aren't using clones for something else), while not infinite should be enough.

No way. Not gonna happen in my atomic scripts. I need speed just as much as I need more memory. In my experience, clones can't interact directly with other clones and use their local variables/lists, let alone interact at all without refreshing turned on so that broadcasts can function smoothly.

If a list is imported from Scratch 1.4 or Scratch 2.0 what happens? This project uses a read-only list of over 500000 items consisting of 13-smooth numbers, and it was imported from Scratch 2.0. Haven't tested write access on lists like that.

Zro716 wrote:

You might say I should use more lists if I want more storage. Problem is, I already have multiple lists holding simple information, and 3.0 simply refuses to allow more lists to be created on demand. There's just no way to get around this unless I build some crazy switch for choosing which list to retrieve info from, which only raises the limit but doesn't remove it outright. Each time I would do this it would severely impact performance and make the project a huge waste of time, both for coding and waiting.

I understand that, at least for previous versions of Scratch, huge lists could crash the application, freeze the computer, and make saving the project difficult or impossible for a variety of reasons. This I know all too well, however, I may add that such a limit restricts the scope of projects by culling out the more advanced kinds like per-pixel image processing, world generation, and 3d engines with high-poly meshes. I highly suggest better provisions and optimizations to minimize the chance of problems occurring.

So a hard limit of 200000 items is not enough. Please, I'm asking for 1 million items, maybe just half that, if removing it isn't an option the ST is willing to consider.

You should try Snap! it os pretty much scratch 1.4 but it has WAY more blocks and advanced block making, it is good for a step up from scratch

staffordboys wrote:

You should try Snap! it os pretty much scratch 1.4 but it has WAY more blocks and advanced block making, it is good for a step up from scratch

I have, it's way too slow to be a replacement.

For some people it could be useful but…
Not for most.

Abitofevrything wrote:

For some people it could be useful but…
Not for most.

A raise to the list size limit wouldn't affect these people in any way.

No support.

There is an workaround to this by switching to a smaller costume and switching back, which is very easy to do.

--Snowball-- wrote:

No support.

There is an workaround to this by switching to a smaller costume and switching back, which is very easy to do.

Read the suggestion carefully — this is referring to the size of a list, not a sprite.

Za-Chary wrote:

--Snowball-- wrote:

No support.

There is an workaround to this by switching to a smaller costume and switching back, which is very easy to do.

Read the suggestion carefully — this is referring to the size of a list, not a sprite.

Oh man, I clearly wasn't paying attention, now I support; an extended list size limit would be useful for many reasons. Sorry for that…

I want something like a 4294967296 item limit, item numbers starting at 0

(length of [Example v] :: list)
      <(4294967295) = (32-bit limit :: #FACE87)?>
say (Example ::list)

What I want is for the list numeral to be 32-bit instead of an artificial limit of 200000
I want to avoid Pi calculators glitching out when the numeral is more than 100000

No support. I don't know a project that needs a 4.3-billion-item list.

If you're going to make a limit that is so high, it might as well be gotten rid of which probably won't happen.

coder2045 wrote:

No support. I don't know a project that needs a 4.3-billion-item list.

If you make a prime sieve to find prime numbers, you need x list items where x is the largest prime you want to find (unless you make a segmented sieve). I made a prime sieve and I got it to calculate 100,000,000 digits in phosphorus. That can't be done anymore in 3.0.

Also, I just remembered why I didn't go to a billion. If creating a list of 1 billion zeros causes Scratch compiled into JavaScript with Phosphorus to run out of memory, then I imagine that you wouldn't really be able to store anything in a list with 4 billion items.

imfh wrote:

If you're going to make a limit that is so high, it might as well be gotten rid of which probably won't happen.

coder2045 wrote:

No support. I don't know a project that needs a 4.3-billion-item list.

If you make a prime sieve to find prime numbers, you need x list items where x is the largest prime you want to find (unless you make a segmented sieve). I made a prime sieve and I got it to calculate 100,000,000 digits in phosphorus. That can't be done anymore in 3.0.

I still don't get why we need an over-4-billion-item list.

coder2045 wrote:

imfh wrote:

If you're going to make a limit that is so high, it might as well be gotten rid of which probably won't happen.

coder2045 wrote:

No support. I don't know a project that needs a 4.3-billion-item list.

If you make a prime sieve to find prime numbers, you need x list items where x is the largest prime you want to find (unless you make a segmented sieve). I made a prime sieve and I got it to calculate 100,000,000 digits in phosphorus. That can't be done anymore in 3.0.

I still don't get why we need an over-4-billion-item list.

Scratch will crash before you can create a list with that many items so there isn't much point.

Pretty sure scratch will crash before you can actually create a list with 4 billion items