I’ve released cattorch, a tool to bridge the gap between PyTorch and Scratch. It transpiles torch.nn.Module objects into standard Scratch sprites using torch.export. Fully supports MLPs and Transformer-based architectures (Linear, ReLU, LayerNorm, Softmax, RoPE, and Causal Masking).

Install: pip install cattorch

Demo & Docs:
PyPI Link | Example Scratch Project

This is really cool! I've had a probably ambitious idea for a while to port roneneldan's TinyStories 1M to Scratch and I'm glad to see a framework for nn's in general being made, looking forward to seeing transformer operations added

also just came across people who have actually made transformer lms on scratch

ajskateboarder wrote:

This is really cool! I've had a probably ambitious idea for a while to port roneneldan's TinyStories 1M to Scratch and I'm glad to see a framework for nn's in general being made, looking forward to seeing transformer operations added

also just came across people who have actually made transformer lms on scratch

Decided to work on this because I actually did make a really small language transformer in scratch recently (9400 params, super small) and wanted to do larger one. Doing all the tensor logic by hand in Scratch takes some time though, so I figured I could just make templates for each operation and have python string them together and fill in constants. High upfront time cost but after that its nearly free to try a new model.

Transformers and LLMs are fully supported now!
https://pypi.org/project/cattorch/

ajskateboarder wrote:

This is really cool! I've had a probably ambitious idea for a while to port roneneldan's TinyStories 1M to Scratch and I'm glad to see a framework for nn's in general being made, looking forward to seeing transformer operations added

also just came across people who have actually made transformer lms on scratch

I looked into trying TinyStories 1M in this as a test, and there are some issues that would make this difficult. First, just pulling from HF won't work, cattorch can't handle some of the tuple and conditional stuff the HF models use, but it is possible to remake the model code without those things. The real problem would be the 3 million embedding params, fitting these into a scratch project would be very difficult. You'd need to aggressively quantize and split them across multiple lists, which cattorch doesn't do automatically.

NormallyNormal wrote:

The real problem would be the 3 million embedding params, fitting these into a scratch project would be very difficult. You'd need to aggressively quantize and split them across multiple lists, which cattorch doesn't do automatically.

i see. i swear i can remember seeing a language model project that used a unique weight compression technique, but can't remember. it wasn't one of the mochi ai projects i mentioned earlier

will try out some models though