I (and a few other users) am working to download the old Scratch Forums, because Lightnin said they need to be closed soon because of security issues. However, after talking with him and telling how to fix the immediate problems, he agreed to keep it up long enough for a HTML only version to be made.

Lightnin said he also may run it locally so that the ST can have their own HTML-only version to store and make public. Anyway, if you want the code, it's available on Github. The instructions are in the description. This is a PHP CLI script, so you need to have PHP installed on your computer (if you're on Linux, get it using apt-get, if you're on Mac OS, use MAMP or it may be built in, and if you're on Windows, get XAMPP and change your PATH settings as necessary).

Or if you're on a non-Ubuntu Linux system (such as Red Hat [Fedora]), use yum.

3DSfan12345 wrote:

Is MAMP command line? I hate command line.

This is a command line application, like it or not.
Command line isn't so bad, as long as you know what you're doing, or only use scripts from sources that you trust.

3DSfan12345 wrote:

Is MAMP command line? I hate command line.

MAMP is mostly graphical. However, for this script, you do need to use the command line.

My archive is well under way. Although it may seem like a while, I think I will have it finished by next Thursday.

@jvvg A neat trick is to randomise the order a bit, and have the script only download the file if it doesn't already exist. That way, you can run several instances of the script in parallel, which makes it a lot faster.

blob8108 wrote:

@jvvg A neat trick is to randomise the order a bit, and have the script only download the file if it doesn't already exist. That way, you can run several instances of the script in parallel, which makes it a lot faster.

However, I intentionally want it not to go too fast, because otherwise my ISP may suspect a DDoS and disconnect me, or even worse report me to legal authorities.

@jvvg It seems unlikely. Maybe it's just my slow internet connection, but even with several running in parallel I still barely managed even 1MB/s.

blob8108 wrote:

@jvvg It seems unlikely. Maybe it's just my slow internet connection, but even with several running in parallel I still barely managed even 1MB/s.

It's not the bandwidth I'm concerned about. It's the massive amount of HTTP requests. If I make a request every second for 10 hours, it looks a lot like a DDoS. Many DDoS attacks use seemingly slow requests but just have them coming from millions of computers at the same time.

I could do the downloading and stuff; I have a good connection.

DigiTechs wrote:

I could do the downloading and stuff; I have a good connection.

For this script, you can do it on a 1 Mb/s connection. It's more about ping, and I have about 33 ms ping. However, you also shouldn't take full advantage of your connection for this, because your provider may suspect you of a DDoS attack and cancel your service.

jvvg wrote:

DigiTechs wrote:

I could do the downloading and stuff; I have a good connection.

For this script, you can do it on a 1 Mb/s connection. It's more about ping, and I have about 33 ms ping. However, you also shouldn't take full advantage of your connection for this, because your provider may suspect you of a DDoS attack and cancel your service.

meh, I actually have done a DoS attack once; to test how strong my internet was. (Yes, it's a denial of service attack if there's only one user, not a distributed one. That's MORE than one )

Now I've only got about 20K topics left to archive. If I keep the program running constantly, it will take about 23 hours to completely finish.

A bunch of files didn't copy correctly, so I have about 15K topics left now. Once that is done, I will have a complete archive.

It is finally done! I now have a complete archive of the old forums (overall, it's about 3.2 GB). If the Scratch Team does close down the archive (which Lightnin said they probably will soon), I will be able to recover all of the old topics for everybody.

Well then…

UPLOAD THEM!

DigiTechs wrote:

Well then…

UPLOAD THEM!

I'd like to, but I don't know of any file host that lets me upload 3.2 GB of HTML data.
For now, if you want your own copy of the archive, you still need to run the script yourself.

I also wrote a simple search tool that lets me search the topic subjects, so if the official archive gets closed down permanently (Lightnin still plans on closing it, but he said that they might use my script to convert everything into HTML pages which are safer) and the HTML version is not used, then I will be able to locate everybody's topics quickly.

Could you compress it and put it on dropbox for people to download?

danwoodski wrote:

Could you compress it and put it on dropbox for people to download?

Compressed, it's still 730 MB. I may do that, though. If I do, it won't be available for a little while.