Scratch, GitHub and conversion from Blockly to text

RokCoder

Edit: The information below gives a good overview but is outdated - see my github repo for up-to-date information and simple steps for using the tool

TLDR;

1) The website allows you to view any Scratch 3.0 project in a text-based language

Fully automated GitHub experience for Scratch projects.

2) Using the given batch file (shown below)

Automatically create and download current SB3 (project) and all scripts for all sprites in text format
Extracts SB3 to sb3 subfolder
Extracts and correctly renames all assets from SB3 file to assets subfolder
Extracts all text format scripts to scripts subfolder
Automatically starts GitHub Desktop, ready for you to enter your commit comment

On large Scratch projects it would be very useful to have some form of source control (Git, etc) to monitor changes and keep safe backups. Unfortunately Scratch SB3 files don't lend themselves well to this so I've come up with an easy-to-use solution.

Run a simple batch file and you will have all the currently used assets as well as the current sb3 and ‘textified’ scripts automatically copied into folders and GitHub Desktop automatically fired up and ready for you to add your commit message.

Note that I'm using the following scenario for my example -

Windows PC (so the batch file approach will need tweaking for other OS)
Using my BBC Computer emulator project whose project id is 515569918
Using GitHub Desktop as my source control (with the basic repository already set up)
I keep my batch file in a bin sub-folder

The first issue is that an SB3 file is essentially a ZIP file and, even if we extract the project.json from within, we still have a file that isn't much use for source control. The first part of this solution is my sb3-to-txt converter at rokcoder.com/sb3-to-txt. From there you can simply enter the project id (515569918 for this example) and see the parsed text files appear separated into one for each sprite.

There's a download button on the site which will save all of those text files to your downloads folder. For automation purposes I've added two optional URL parameters to the site - projectId= and autosave.

If you use rokcoder.com/sb3-to-txt?projectId=515569918&autosave you'll see that the specified project is parsed and downloaded automatically without any user intervention (alongside the constructed SB3 file). This gives us files which are definitely usable for source control so next comes the part of making it quick and easy to commit any changes. I use the following batch file which I name commit.bat and keep in a sub-folder called bin.

@echo off
setlocal
set /A PROJECT=515569918
set DOWNLOAD=%USERPROFILE%\Downloads
del "%DOWNLOAD%\%PROJECT%.zip"
rem start "" "http://www.rokcoder.com/sb3-to-txt?projectId=%PROJECT%&autosave"
start "" "http://localhost:3000?projectId=%PROJECT%&autosave"
call :clean scripts
call :clean sb3
call :clean assets
:wait
if exist "%DOWNLOAD%\%PROJECT%.zip" goto ready
timeout 1 /nobreak > nul
goto wait
:ready
tar -zxf %DOWNLOAD%\%PROJECT%.zip -C %~dp0..
cd %~dp0..\assets\
tar -zxf %~dp0..\sb3\%PROJECT%.sb3 -C . *.mp3 *.wav *.png *.jpg *.svg *.bmp *.jpeg *.gif > nul
for /F "usebackq delims=" %%a in ("assetFolders.txt") do mkdir %%a > nul
for /F "usebackq delims=" %%a in ("assoc.txt") do move %%a > nul
del assocFolders.txt
del assoc.txt
github %~dp0..
exit /b
:clean
if not exist %~dp0..\%1 mkdir %~dp0..\%1
del /Q %~dp0..\%1 del %~dp0..\%1\*.* > nul
exit /b

You need to set the PROJECT variable up with the project ID that you're working on. This could be passed into the batch file as a parameter but it's a cleaner and quicker solution this way as every project will have its own batch file anyway. What this batch file does is to extract everything from the downloaded zip file into the best arrangement for source control (GitHub). The scripts will be put into a scripts subfolder, the assets until an assets subfolder and the SB3 into an sb3 subfolder.

Now, whenever you want to commit you simply run the commit.bat batch file. You can double-click it from a file explorer, run it from the commend line, set a shortcut to the file from your desktop, etc. All the work is done for you and it'll drop you into Github Desktop, ready to add your commit comment.

Miscellaneous notes

The batch file will remove all old files from the assets, sb3 and scripts folders. My local repo folder structure is generally similar to this -

Project
|—> bin (where the batch file and any other tools are stored)
|—> assets (automatically created and filled by the batch file)
|—> scripts (automatically created and filled by the batch file)
|—> sb3 (automatically created and filled by the batch file)
|—> srcGraphics (for GIMP-format layered graphics which are processed for the project)
|—> srcSounds (for pre-edited original sound and music files)

Musings and possibilities

Nothing much to muse at the moment as this works very well! That said, let me know of any projects it fails on and I'll fix it promptly.

Last edited by RokCoder (Dec. 9, 2021 08:34:30)

gosoccerboy5

Oh, nice. I'll look a bit more into this when I have the time (not like I'll understand much of it XD)

kriblo_test

RokCoder wrote:
Musings and possibilities

It looks so nice! In fact, let me know when you've implemented the Text to Blocky part. Maybe I can see the light at the end of this carpal tunnel

RokCoder

kriblo_test wrote:
…when you've implemented the Text to Blocky part.

I don't recall mentioning that at all… ever! It sounds like a syntactical-checking nightmare of a rabbit hole. And yet it's somehow such a strangely appealing idea! Why must you come up with such temptations?

ninjaMAR

Very cool! I got to try

gdpr5b78aa4361827f5c2a08d700

Interesting

samq64

Project ID 384515871 (Blue OS) doesn't work.

RokCoder wrote:
kriblo_test wrote:
…when you've implemented the Text to Blocky part.
I don't recall mentioning that at all… ever! It sounds like a syntactical-checking nightmare of a rabbit hole. And yet it's somehow such a strangely appealing idea! Why must you come up with such temptations?

Last edited by samq64 (April 30, 2021 22:53:49)

9gr

samq64 wrote:
Project ID 384515871 (Blue OS) doesn't work.
RokCoder wrote:
kriblo_test wrote:
…when you've implemented the Text to Blocky part.
I don't recall mentioning that at all… ever! It sounds like a syntactical-checking nightmare of a rabbit hole. And yet it's somehow such a strangely appealing idea! Why must you come up with such temptations?

appel doesn't work either.

could you make a script for bash?

Barthdry

I think this isn't stable yet and has some problems in some of my projects due to some reason. It has a lots of Bugs. But you have tried it well or the one who made it tried well

Last edited by Barthdry (May 1, 2021 13:05:02)

RokCoder

samq64 wrote:
Project ID 384515871 (Blue OS) doesn't work.

Works now - thanks for letting me know. I wasn't handling (pointless) things like repeat until loops that contained no conditional whatsoever. Please let me know if you come across any other projects that don't work.

Last edited by RokCoder (May 1, 2021 19:38:29)

RokCoder

9gr wrote:
appel doesn't work either.

Works now - thanks for letting me know. Was the same fix as required for the previous comment.

9gr wrote:
could you make a script for bash?

Not something I'm as familiar with so will require some Googling. I'll likely do it when I'm next working on a Mac but can't say when that will be.

RokCoder

Barthdry wrote:
I think this isn't stable yet and has some problems in some of my projects due to some reason. It has a lots of Bugs. But you have tried it well or the one who made it tried well

It's stable for my projects but is really in a beta stage. It would be helpful if you give me the project numbers for any projects that aren't working with it. I've added several fixes already today so you may find they work now.

RokCoder

Check the first post for massive updates!

The automated experience now includes the scripts (in text format) as well as all the assets and the sb3 file. All of these are built and/or extracted automatically by the batch file. Your GitHub will be able to track all changes in code or assets as well as offering safe backups of your full sb3 at the click of a button!

ArnoHu

Thanks a lot for this very helpful tool, and for already having picked up some suggestions. I had used s3blocks so far to commit my Scratch projects to github, but am currently switching over.

Some additional findings and suggestions:

I have a lot of lists in several projects, which are filled dynamically. If I don't clear them all, the lists take by far most space within the source code. One reason being they use one row per list item. Maybe we could comma-separate them?
Some lists appear to have items, although I have cleared already. But maybe this is due to some Scratch API or caching issue.
We talked about this before, but I think readability might be better for list- and other operations by an OOP-like approach such as
```
MovesIdx.ReplaceItem((Depth + 1), "1")
```
or even
```
MovesIdx.ReplaceItem(Depth + 1, 1)
```
instead of
```
List.ReplaceItem(MovesIdx, (Depth + 1), "1");
```
Basically using the “this” reference name as a prefix. Maybe it's possible to provide export option to select the output format depending on personal preference? I also just notice ReplaceItem() might be a bad example, because IMHO
```
MovesIdx[Depth + 1] = "1"
```
is the best option here.
WhenKeyPressed() event handlers appear under orphaned blocks.
Numeric constants are always put under double quotes. Now I know this might be a tough one due to Scratch's variant approach - how to distinguish between numeric values and strings. Maybe detect whether the value fits into a IEEE 754 double precision variable, I think that's the way Scratch does it as well.
Could we comma-separate custom block parameters for better readability?
Expressions passed as custom block parameters are always put into brackets, even they are simple (no expression tree, also see sample above)

Last edited by ArnoHu (May 2, 2021 16:58:53)

Chiroyce

Woah this is cool!! Very nice, it will be very useful for debugging variables quickly

RokCoder

Lots of fantastic feedback and suggestions to take on board. Thanks!

ArnoHu wrote:
I have a lot of lists in several projects, which are filled dynamically. If I don't clear them all, the lists take by far most space within the source code. One reason being they use one row per list item. Maybe we could comma-separate them?

I'm going to leave this for the simple reason that it helps remind you to remove redundant list entries! Most large lists are going to be dynamically created and it serves no purpose to have their contents in source control.

ArnoHu wrote:
Some lists appear to have items, although I have cleared already. But maybe this is due to some Scratch API or caching issue.

That's really odd. I always have a ‘thumbnail’ broadcast receivers in my sprites which clears any redundant data (and displays the thumbnail) which I call before saving. I've noticed it sometimes doesn't clear everything and the first call which is odd. Always seems to work on the second or third though. This kind of ties in with the last point about it being useful to see if they've not been cleared properly.

ArnoHu wrote:
We talked about this before, but I think readability might be better for list- and other operations by an OOP-like approach such as
MovesIdx.ReplaceItem((Depth + 1), "1")
or even
MovesIdx.ReplaceItem(Depth + 1, 1)
instead of
List.ReplaceItem(MovesIdx, (Depth + 1), "1");
Basically using the “this” object name as a prefix. Maybe it's possible to provide export option to select the output format depending on personal preference? I also just notice ReplaceItem() might be a bad example, because IMHO
MovesIdx[Depth + 1] = "1"
is the best option here.

I like the idea of the export option. I'll add then when I have a little more time. It'll likely be a URL parameter option.

ArnoHu wrote:
WhenKeyPressed() event handlers appear under orphaned blocks.

Fixed

ArnoHu wrote:
Numeric constants are always put under double quotes. Now I know this might be a tough one due to Scratch's variant approach - how to distinguish between numeric values and strings. Maybe detect whether the value fits into a IEEE 754 double precision variable, I think that's the way Scratch does it as well.

Done

ArnoHu wrote:
Could we comma-separate custom block parameters for better readability?
Expressions passed as custom block parameters are always put into brackets, even they are simple (no expression tree, also see sample above)

I don't much like the current approach but it's difficult as custom blocks can contain labels such as place (object) at x: (x), y: (y) where comma separating doesn't really work.

MikeDEV

Is there a way to commit an sb3 file locally instead of downloading the sb3 from Scratch?

RokCoder

MikeDEV wrote:
Is there a way to commit an sb3 file locally instead of downloading the sb3 from Scratch?

Not in the current set-up. You would have to upload your local file to a Scratch project. The Scratch project doesn't need to be shared though.

CST1229

I think the batch file could be replaced with a node.js script, which cuts out the need for a browser and downloading a file, is cross-platform and is faster because of the lack of need for a browser.

RokCoder

CST1229 wrote:
I think the batch file could be replaced with a node.js script, which cuts out the need for a browser and downloading a file, is cross-platform and is faster because of the lack of need for a browser.

I've zero experience with node.js as I tend to use php for my own websites so would need some more details for this. I assume you'd need to install node.js and run it via localhost or similar? I'd be interested to hear your thoughts on how this might work. I'm fine with the batch file for myself as I'm not sure anyone else uses this for source control. That said, it's a clunky solution and is possibly a barrier that prevents other people from using it (and I find it essential these days when working on a Scratch project of any reasonable size).

The batch file shown on this page is very outdated - the current one can be found here and you'll also find simpler usage steps in the readme of that repo.

To me there seem to be various areas which might be problematic to creating an out-of-the-box cross-platform solution but I would be very happy if you want to have a go at a node.js approach!

Discuss Scratch