Something is rotten in the state of hyperlinks

Posted on 5 mins read View source

A couple of months after I started doing the monthly media diet posts I ran into an annoying situation: One of the links I had collected in my notes throughout the month was no longer accessible when I sat down to write the post. (I usually only keep brief notes throughout the month, and then spend a weekend or two at the end to sort and flesh out my thoughts.) And I don’t just mean that it got pay-walled since I had read/found it – the author had fully yanked it from the web. Even the Wayback Machine or archive sites did not have a copy yet. I was pretty bummed out – not necessarily because it’s was super important to share that particular article with you (I actually doubt many readers even click through to a lot of my links), but now I couldn’t preserve that part of my thoughts anymore. The media diet is largely also for myself, especially details like the links. I want to be able to revisit them in the future.

To that end, I have been running a Readeck instance for about a year now. Initially I just aimed to develop the habit of archiving anything interesting I read, but of course it didn’t take long until I realized I should automatize this some more. Why risk ever forgetting about a link? The way I’ve implemented this is quite boring: Yet another script in the ever-growing collection I’m using for this blog!

The archive-links.py script grabs all URLs from either:

  • a single Markdown file, or
  • an entire directory of Markdown files.

For each URL it then:

  1. Compares it against an ignore list (e.g. I don’t need to archive an annoying.technology link – I have the source files for that blog).
  2. Skips any URLs that have been archived already.
  3. Sends any new “regular” URLs to Readeck.
  4. Sends any new video URLs to yt-dlp (which I’ve written about before).

Video

I have a rather painful example for why I’m also archiving videos that I link to: In this post I painstakingly collected the exact clips on the Comedy Central website that would let you re-experience the magnificent Carlos Danger saga during in the Summer of John. This was actually already the second time I did that work. The website and the videos’ URLs had changed since I collected them the first time a decade ago, but I was pleasantly surprised they were still available. Unfortunately, that changed about a year ago. If I had already built and been using this script before then, I’d now still have a copy of those clips.

The storage requirements for video aren’t as bad as you might think: I linked to over 120 videos with a total runtime of just over 24 hours – and the archived copies only take up 16GB. Well worth it even in this terrible recession and hardware crisis. It’s insignificant enough that I never think twice about it: “Hm, this is just a song I also have in my music library – do I really need to waste space on the music video for that?” is not a question I want to be asking myself. Just archive it. You never know when you’ll be happy you did.

Whenever I publish a new blog post, I manually run the script afterwards to archive new links. But I also did one big retroactive run for all my old posts. The amount of link rot in these old posts was awful – you can see in this commit how much I had to fix.

A lot of articles moved behind paywalls, so Readeck can’t ingest them automatically. I’ve been using a fairly simple workaround for that – I just open two tabs: the regular page that shows me the paywall, and an archive site version to go around the paywall. Then I’ll use the browser’s inspector to copy the entire <body> (or main <article> or similarly suitable) element from the archived copy and paste it into the other tab’s body. This will usually look pretty bad in the rendered view of the browser, but that’s okay – due to how the Readeck browser extension works, this Frankenstein’d page can usually be cleanly ingested. The result is a Readeck entry that references the correct original URL (and not the archive site URL), but with the full article content correctly extracted and retained in my personal archive forever.

Next steps

I’m not archiving every website I link to. It’s mostly focused on articles, as that is what Readeck is for. Something like a web shop’s product page doesn’t really fit into that approach, for example. For now, that seems like an okay trade-off. Similarly, keeping copies of Wikipedia pages also seemed irrelevant so far – but given the increasing attacks on any remaining slivers of good on the internet, I’m re-evaluating that stance. It might be worth it to archive the exact version of a Wikipedia page at the time I’m linking to it. In theory that should be something that can easily be reconstructed through the pages’ edit history, but we’ve now seen often enough how quickly things taken for granted can ruthlessly be yanked away.

Besides widening the scope of what gets archived, automating the process even further (likely via continuous integration) is what I’d like to explore next.