Discuss Scratch

blob8108
Scratcher
1000+ posts

Forum Archive

I made a temporary mirror of the now-broken forum archives:

http://scratchforums.blob8108.net/forums/
All existing links work: given any link to an old forum page, simply replace “archive.scratch.mit.edu” with “scratchforums.blob8108.net”.

I don't know how long I'll keep it online. But for now, enjoy!


________________________

PS. If you like, you can make your computer use my mirror instead of the archives. You can do this by adding the following line to your `/etc/hosts` file:

188.226.155.121 archive.scratch.mit.edu

Then when you visit http://archive.scratch.mit.edu/, you'll really be visiting my mirror.

You'll have to remember to undo this when I turn off my server or the ST fix the official archive.

Last edited by blob8108 (May 8, 2014 19:09:33)


tosh · slowly becoming a grown-up adult and very confused about it
djdolphin
Scratcher
1000+ posts

Forum Archive

Gah, you beat me to this. I've been trying to upload 3 gigabytes of files for months. I have a copy of the archive running locally on my computer. Anyway, great job!

Last edited by djdolphin (May 8, 2014 22:16:44)


!
blob8108
Scratcher
1000+ posts

Forum Archive

djdolphin wrote:

Anyway, great job!
Thanks!
I've been trying to upload 3 gigabytes of files for months.
The hard part is getting things like post redirects to work properly. I'm serving most things as static files, with a bit of Python to do post redirects (so links like viewtopic.php?pid=… still work). The post -> topic mapping is looked up in a database. You really don't want to store each of the 1.4 million posts as a static file. (Since each file takes up at minimum the filesystem's block allocation size – which is usually 1024 bytes.)

The topic pages still take up most of the space (~4 GB). I could probably reduce it further by storing those in a database, too, but that gets messy.

tosh · slowly becoming a grown-up adult and very confused about it
jvvg
Scratcher
1000+ posts

Forum Archive

Wow, cool! Where are you hosting it?

jvvg
Administrator and backend engineer on the Scratch Wiki
Executive director of FutureSight Technologies
blob8108
Scratcher
1000+ posts

Forum Archive

jvvg wrote:

Wow, cool! Where are you hosting it?
On a $5/mo server from http://www.digitalocean.com. They were giving away free $10 codes…

tosh · slowly becoming a grown-up adult and very confused about it
Swampert11
Scratcher
1000+ posts

Forum Archive

Thank you for posting this!

Do you know a good project that doesn't have a lot of attention? Suggest a project to be put in this studio that is unpopular.
___________________________________________________________________________________

Only a Scratcher <——– Please look at it!
___________________________________________________________________________________

;












davidkt
Scratcher
1000+ posts

Forum Archive

Or, you could just use the Wayback Machine…

Remember when I looked like this? I still do.


Float, my Scratch 2.0 mod | My (somewhat under-construction) blog
blob8108
Scratcher
1000+ posts

Forum Archive

davidkt wrote:

Or, you could just use the Wayback Machine…
Sadly, most of the forums aren't in their archive.

tosh · slowly becoming a grown-up adult and very confused about it
derpmeup
Scratcher
1000+ posts

Forum Archive

davidkt wrote:

Or, you could just use the Wayback Machine…
The Wayback Machine doesn't have every topic archived.

hey bro that's a pretty good meme xD!
jji7skyline
Scratcher
1000+ posts

Forum Archive

Thanks so much for doing this, it's invaluable! I hope this stays up forever!
blob8108
Scratcher
1000+ posts

Forum Archive

jji7skyline wrote:

Thanks so much for doing this, it's invaluable! I hope this stays up forever!
Heh Hosting is a little expensive for a penniless student… I need to sort out giving the ST a copy that works, but making a proper static one is a little tricky

tosh · slowly becoming a grown-up adult and very confused about it
mighto360
Scratcher
10 posts

Forum Archive

yeah


comp09
Scratcher
1000+ posts

Forum Archive

blob8108 wrote:

jji7skyline wrote:

Thanks so much for doing this, it's invaluable! I hope this stays up forever!
Heh Hosting is a little expensive for a penniless student… I need to sort out giving the ST a copy that works, but making a proper static one is a little tricky

Are you okay with me pointing my wget request machine-gun at your server to static-ize the website?



Visit the website of Andrew Sun!


djdolphin
Scratcher
1000+ posts

Forum Archive

comp09 wrote:

blob8108 wrote:

jji7skyline wrote:

Thanks so much for doing this, it's invaluable! I hope this stays up forever!
Heh Hosting is a little expensive for a penniless student… I need to sort out giving the ST a copy that works, but making a proper static one is a little tricky

Are you okay with me pointing my wget request machine-gun at your server to static-ize the website?
Isn't it already static?

!
comp09
Scratcher
1000+ posts

Forum Archive

djdolphin wrote:

comp09 wrote:

Are you okay with me pointing my wget request machine-gun at your server to static-ize the website?
Isn't it already static?


blob8108 wrote:

djdolphin wrote:

Anyway, great job!
Thanks!
I've been trying to upload 3 gigabytes of files for months.
The hard part is getting things like post redirects to work properly. I'm serving most things as static files, with a bit of Python to do post redirects (so links like viewtopic.php?pid=… still work). The post -> topic mapping is looked up in a database. You really don't want to store each of the 1.4 million posts as a static file. (Since each file takes up at minimum the filesystem's block allocation size – which is usually 1024 bytes.)

The topic pages still take up most of the space (~4 GB). I could probably reduce it further by storing those in a database, too, but that gets messy.

Static-izing the website would make it easier to host on GitHub Pages, though. The post id redirections could be done with a bit of JavaScript hackery.

As a side note, could blob8108 possibly make the database/files available for us? You could 7-zip everything up and upload it to GitHub releases…

Last edited by comp09 (Feb. 27, 2015 21:50:14)




Visit the website of Andrew Sun!


blob8108
Scratcher
1000+ posts

Forum Archive

comp09 wrote:

The post id redirections could be done with a bit of JavaScript hackery.
Looked into that. Not feasible.

tosh · slowly becoming a grown-up adult and very confused about it
nXIII
Scratcher
1000+ posts

Forum Archive

blob8108 wrote:

comp09 wrote:

The post id redirections could be done with a bit of JavaScript hackery.
Looked into that. Not feasible.
Couldn't you generate a global post ID → topic ID/page JSON file and then just have viewtopic.php redirect if it gets a post ID?

Last edited by nXIII (Feb. 28, 2015 01:26:12)


nXIII · GitHub
comp09
Scratcher
1000+ posts

Forum Archive

blob8108 wrote:

comp09 wrote:

The post id redirections could be done with a bit of JavaScript hackery.
Looked into that. Not feasible.

You could generate a couple thousand files, each with 1000 topic/page numbers in it. Each file should only be several kilobytes.

For example, to find where the post 1,556,664 is, a script could request the file 1556.txt, which would contain something like the following:
180110,2
162848,1
184942,5
108709,4
111153,7
181590,11
114507,9
117521,1
124810,6
114089,8
...
and so on. The script would go to line 664, which would have “117391,5”, which tells it to go to /forums/viewtopic.php?id=117391&p=5#p1556664.

Sounds feasible to me.

Last edited by comp09 (Feb. 28, 2015 02:26:59)




Visit the website of Andrew Sun!


blob8108
Scratcher
1000+ posts

Forum Archive

I already thought of that.

tosh · slowly becoming a grown-up adult and very confused about it

Powered by DjangoBB

Standard | Mobile