Wholeheartedly agree. It's sad to slowly see the bitrot in my own bookmarks, trying to pull up important resources or media from only a few years ago only to hit a 404 or domain placeholder. And many times there is nothing on Archive.org or similar.

That ArchiveWeb extension looks excellent for custom captures. Picking the pages and elements you want during capture is a great feature.

For quicker one-offs, SingleFile gives you a single HTML with all the media of a page: https://www.getsinglefile.com

If you're making archiving a more involved part of your workflow, I highly recommend ArchiveBox. It can save pretty much every type of file from everywhere, in almost every format you can imagine, while building a searchable index of the archives. Been using it successfully for many years: https://archivebox.io

one of the great tragedies of my life was when https://buthowdoitknow.com/ went offline, taking the visual emulator with it

i managed to reconstruct (and then improve) it with an LLM:

https://scott-cpu.cs.montana.edu/

Over the summer, I had the urge to do this with a few favorite blogs. wget did the trick for static content

Another great writeup about backing up websites: https://gwern.net/archiving

https://hister.org/ is a neat website indexing and archival tool.

I recently spent a few days trying to figure out and discover my old websites and things from the early 2000s on the web archive. I hadn't realised my memory had gone quite so bad, but it was revitalising to find some of what I'd forgotten. So much of what I do remember is long dead and missing, sadly.

I thought it might be rose-tinted glasses, thinking that the earlier internet was better, but spending time retracing my teenage steps made me even more certain that it was indeed better. So many derpy little websites made by nerds, so many fun games and websites that weren't preying on vulnerable people, and everything spread out across many places - not centralised in the hands of large corporations. I've been using LLMs to have fun recreating concepts from that time, like my own forum software, as some sort of anti-progress. It's been fun.

In 5 years, cloud LLMs may have largely replaced the open Internet as a source of information, with centralized control and ability to rewrite the past.

Please, please buy books and keep them in good condition, and please archive websites with important information (assuming it's legal in your jurisdiction). There are still plenty of excellent sites and blogs around from the 2000s and even earlier, but they won't be there forever. Ironically, coding agents have made archival much easier -- I've always found problems with using pre-written tools, but Claude or Qwen make it trivial to download, test, and archive sites on your local.

Have a look at Hister [1], a 'personal web history search engine' which by way of extensions for browsers captures your web history and stores actual page content on your own instance where it can be browsed and searched. It does not store images so if that's what you're mostly interested in it won't work for you. For text is works quite well, maybe sometimes too well since it also stores ephemeral things like HN comment pages unless you tell it not to.

[1] https://github.com/asciimoo/hister/

the last source of truth are forums.. but not reddit... if it has a CEO it will rot and enshitify.

I'm talking about forums... the bmw forum for your old car... the motorcycle forum for the enthusiast of the SV650... those places are full of anecdotes, people talking, lending a hand to another person... being humans and care...

Which is what social media is/should/was about, is what internet is about before the enshitification happened... I would vote to have this ID associated to you via check of your local goverment issued id and ONLY UPPON VERIFICATION of you being using that you can log and read and use those websites without tracking you (Yes 10,000 ifs where this is secure, handled in a confidential matter, the whole tech is not to track that you are doing something... but that a human is behind that, no api, no robot, no adds, no nothing... just the old school forum where you can go, talk, submit a photo... where someone who wanted to spend time to write something did it, SO WE CAN ENSURE TWO THINGS, A HUMAN IS THERE, AND NO FVK!NG CEO WILL ENSHITIFY IT, no to track people, but to know that a person is interacting there)

But because "empathy is the weakness" and shareholder value...

Yes, and with AI you can even back up the logic behind a website (the back-end), and even generate fake but plausible data to fill it if crawling is too much work.

This is why I started using Obsidian Web Clipper. There is SOME work to learn it and set it up, but the small amount of upfront work will pay for years to come. And the folder itself is backed by Dropbox, so I don't have to worry about losing it all forever.

Ages ago (like 2001 or so?) I started building something to automatically archive everything I browse. I used tcpflow (https://www.circlemud.org/jelson/software/tcpflow/). I've seen a fair bit of varions of this idea over the years

Then SSL/TLS started getting widespread and I lost focus. Nowadays, I guess you would build it as an extension in the browser.

I did actually. I took a whole day to go to each of my individual saved bookmarks and save the articles as pdf using edge reader mode, sometimes hunting down articles on the way back machine

i'm saddened that http://macsecret.com/ along with /files directory seems to have all be been lost. I has waiting on some storage for the NAS to archive it and it went away before I could back it up.

I learned this the hard way by getting too casual with Angelfire keeping an old website running for 20 years after I stopped updating it.

It had lots of photos that I hand-scanned from printed film, back in the day when it took a good bit of time to scan photos manually.

The site was a running joke, where I would open it to show people that it still existed... until it didn't, and I couldn't find my backup of all of the photos. The archive sites have the 'bones' of the website, but not the photos. It was easily avoidable, which is why it is painful.

I tend to save things I read as PDFs because it feels the most robust archival formal for the future.

Bonus suggestion: get that internet archive extension for your browser that automatically sends every page you visit to the IA (if you're ok with the privacy side).

You don't have to worry about having your copy easily accessible, and as a bonus it's also available for everyone else in the world.

Same goes for youtube vids

Thanks for the guide ,I never new about this

There are far better tools that are not built by this person’s company. Dig around and you’ll find an incredible array. Not everything has to be a sales pitch.

I did this for a while. I went so far as to built a highly advanced web crawler webapp that was better than the Internet Archive in a lot of ways.

But the modern web has made this unreasonable. Every webpage is 99% junk (CSS, JS, HTML structure, tracking pixels, UI images, navigation links, cryptominers (I joke)) and 1% content. You need a smart AI system to pick out the good parts, otherwise you'll quickly find yourself schlepping hundreds of gigs of junk to just preserve your favorite websites.

The better idea is: if you love information, back it up. Collect the knowledge from the website, not the website. And then preserve it. Create a free ebook. Or sell it on Amazon kindle. That'll last a lot longer than the website will, and any HTML backup of the site you could possibly make.

I had a graphing calculator website that had 250K monthly active users, from the logs it seemed mostly schools' math classes in Latin America. It was running on a single AWS EC2 Classic.

Ironically, when I started working for AWS, I always felt threatened with sticks and never had space to maintain my own projects. When they deprecated EC2 Classic, my site went down. I didn't get time or mental space to migrate it.

Cest la vie I guess.

I've backed up lots of CS courses, you know those old html websites written by the professors by hand. `wget -mkpnp` works very well for those. Even years after finishing uni, I've used those websites for studying for interviews. Too bad most of them are now gone.

I scrape daily. YouTube, archive,org (sadly getting harder and harder since you have to "checkout" books), worldradiohistory.com…

(Now if hard drives had not become so damn expensive—it's getting harder to be a digital hoarder like myself.)

So, this is a fine idea and all, but aren't the "piss off AI bot" mitigations going to throw a wrench into folks wanting to back up a website?

I guess the web plugins could ideally get around it, but what about the CLI and app solutions?

I know yt-dlp has gone to great lengths to work this problem.

No one uses pinboard to archive pages anymore?

I just use a simple approach - print-to-pdf

Zotero (Desktop) backs up web pages using SingleFile under the hood.

I've tried almost all the popular and many not-popular options to no avail. Does anybody know of an archiver of any variety (Firefox extension, or CLI give-it-a-url, or anything else) that can archive a modern Reddit page with the voting (e.g. I can later see if I up- or down-voted a post)?

Print the digital things you want to last.

God forbid we have an AI-facilitated full Cyberpunk/.hack-style crash where everything gets wiped, the only things that will survive will be on paper. This is the recommendation I give my friends who are digital artists/writers; have a physical record of your efforts beyond what you want to sell.

This is on top of keeping your own digital archives. I'm still haunted by the deviantArt submission I didn't save. The creator took the piece down, then passed away a few years later. A Wayback Machine archive of his userpage with the piece as his featured upload still exists, but the image itself wasn't archived; it just shows a broken image.

I guess I am one of the dinosaurs here if I still use HTTrack for such things?

https://www.httrack.com/

this is why i built linkwarden: https://linkwarden.app

(p.s. it's open source!)

Thought I’d see this noted in comments. Monolith is another solid option: https://github.com/Y2Z/monolith

This is timely for me, since a web forum I care about went offline, and I simultaneously discovered the Wayback Machine for whatever reason never crawled most of it. I asked Claude to try to put together a static site of what could be salvaged from various archives and other sources, but as miraculous as what AI can do these days, it can't restore data that is truly gone - so what it was able to salvage is better than nothing but still missing a lot.

Agree, I backed up Fravia's website back in the day, and it is fun to come back to.

I downloaded a ton of image in the days following 9/11, and burned them to a CD.

I wonder how many other people have archives like that...?

This has come up for me very recently with the multi-week AO3 outage. I'm not a huge user of that site, but it specifically existed as a comprehensive archive of fanworks and of many prior fanfiction sites that have gone offline. I really respected the Organization for Transformative Works as a non-profit with a very targeted mission of archiving these sorts of works and keeping them safe from linkrot and the algorithmic web. Now, after a security incident (which I am assuming is related to the recent LLM-driven discoveries of unpatched vulns in open source hosting software), the whole archive is offline for the next few weeks.

I'd always assumed that OTW would keep AO3 online until the end of time. It, somewhat like this site, was one remaining gasp of Web 2.0 before the algorithmic feeds took over.

Is this not what the internet archive is for?

Video backups are also important too as they can be easily censored when within a walled garden.

Interestingly, "there is no single direct 1:1 Linux clone of NirSoft’s VideoCacheView that automatically monitors browser cache folders via a graphical interface. Released 2007." https://videocacheview.en.lo4d.com/windows (Gemini)

Yup! especially as we are using various ai platforms and ai agents. I created my own artifact aggregator service which hosts the artifacts. I run it on oracle cloud for free and back them up on my gmail drive. https://simple-host.com, and you can back

Obligatory plea: when scraping sites for archive purposes, take care to do so slowly so as to not impose an unfriendly amount of load on the target site. Otherwise you risk becoming part of the problem that is making some take down their resources ATM.

Something I have been doing is saving PDFs of interesting articles. I actually go back for some of them. Organization is always an issue but mostly desktop search finds it.

Agreed. I won a major (at least within the college) music competition at a local college like two decades ago, and now all traces of it have vanished. If I had the URL I might be able to pull it up on the Internet Archive.

Is https://pinboard.in still a thing? I used to be a subscriber but completely forgot about it.

A website I miss was a bullshit corporate jargon generator from Scott Adams. There were 3 columns with corporate jargon. And every time you refreshed the page, a new 3-word phrase would be highlighted. They were the most ridiculous phrases. But one could imagine them being used.

Recursive wget isn't going to pass Anubis.

Agree with this. I started down learning more about archival while I was adding it to my own thing [1]. I wish there was more of an existing toolset for diffing archivals as well - what changed is just as interesting as what is there. Equally, something I kind of think is lacking in WARC/WACZ is exploration of the archives - maybe forensics is too strong a word but it would be nice to have an easier way to explore what you’ve captured and understand the data in different dimensions.

I’ve supported some of this in my own tool, ZShot - capture in sqlite, Quicklook extensions for WARC and WACZ. Just trying to make archiving easy and interpretable.

It’s also been interesting to see people use archival but for different uses: Kiwix [2] showed up in my reading recently, “archival but for offline internet”, which I’m kind of fascinated by.

[1]: https://zshot-app.com/for/archivists/

[2]: https://get.kiwix.org/en/

Any tips on backing up a forum hosted on XenForo?

Or don’t. Lots of things I loved in the past and spent time backing up and backing up the backups that after 15-25 years I realized was a waste of time. Thought I had plans for them but never looked at them. Never referred to them. Finially realized I never was and got rid of the final backups

I agree sorta, but maybe its best to just let things go.

Back it up, or even better, re-implement it to look better than the original!

I did this for the category theory / math website called ncatlab / nlab

https://github.com/taliesinb/nlab

I remember an old niche website I saw some 20 years ago. The whole content was dynamically loaded. If you disabled javascript, you were greeted with a static page that accused you of theft. Attempting to select and copy text or right click resulted in a popup window saying the same.

Nowadays you can just slap your site behind Cloudflare, I guess.

Look what happened to motherless without backups /s

I've started to use a custom bookmarklet for this. If I like something I read or watched, I 'pick' it with the bookmarklet, add my thoughts, and categorise it. The script then checks if it's on the Internet Archive, if not it tries to archive the site, and then save it to my public repo, which then my personal website shows under /library.

Repo link, if anyone is interested: https://github.com/karthikeyankc/picks

What are the legal implications of something like this? What if the site has copyrighted images or videos?

Any simple way to archive an online database, think imdb, but for other interests?

Looks like the ArchiveWeb Chrome extension is no longer accessible. I am getting a 404 error. Seems like the Wayback Machine extension works though: https://chromewebstore.google.com/detail/wayback-machine/fpn...

This made me realize that I don't actually love any website like that. Wikipedia comes the closest, but even if that would disappear, surely someone would have a dump, which I'd grab at that point. Until then, I won't do this.

The two things I go back to are music, and books. Even books not so much, but songs tickle a very specific part of my brain, that I want to reproduce over and over again, and then not for years, and then again. And when I want to go back and I can't, and I know I could, that really hurts, and I feel betrayed. So I do actually archive the music, even from streaming sites.

I'm still rocking my first article online since 2003: https://paradies.jeena.net/artikel/webdesign

At the moment I use Obsidian and some CLI magic to extract my discoveries out of the Web into private notes. I call that My Own Private Wikipedia. Sounds like a good practice as things are organize the way I want, and I can rearrange in my own way and things are backuped as I wish.

Why not use wget to back up website?

I've been using self-hosted Linkding for about a year now to save bookmarks. It will also use a Chromium instance to scrape and archive the content/page in an HTML format.

I've not had to use that yet, but it's also nice to save terms and conditions and the like when you agree to them so there's no sneaky edit.

Ironically the author's website has a copyright notice at the bottom, which means I can't archive it. Right?

It is kind of sad to see how many webpages vanished.

This includes old games. Many vanished. Some made a cameo, coming back, but it is often no longer the same. Case in point: The Reincarnation (TR), as a follow-up to the old Mari Archmage (AM) game. It was great in the late 1990s. Mari went bankrupt, TR came about and improved on many things, but also made a few things worse. Player population declined, then they had that genius "idea" to add bots. That was when I retired. I can not want to be bothered to "interact" with code.

After losing the Japanese indie web of the 00s/10s several times over, I wish it was possible to save not just the websites you love, but the websites you'd love if you actually knew about them. So often do I find out that what I was looking for did exist, but doesn't exist anymore. In particular when only the home page got archived, but not the rest of the website, nor the uploaded files.

It's not really clear to me why people are proposing so many complex approaches or third-party solutions here. What exactly am I missing, what's insufficient about my browser's "Save page As... -> Web Page, complete" functionality?

Man, I wish I started doing this years ago. There are around five websites that I still think about that no longer exist. And no longer have an archive anywhere.

These days I download any site I like.