The web archival site archive.today has become a popular way to bypass the paywalls of news outlets. But how exactly does it do that? Surprisingly, no one knows for sure.

Before we get into the details, let’s clarify an important distinction, because people often mix this up. There is the Internet Archive, a legitimate US-based non-profit organization that offers the Wayback Machine among other services, and then there is archive.today (or archive.is, archive.ph, archive.md, …). Not much is known about its owner, its finances, or pretty much anything else – except that it is unique in its ability to bypass paywalls.

This article is solely about archive.today, henceforth referred to as “The Archive”.

Note that there is an interesting article investigating the founder’s identity and the site’s infrastructure, which led to some serious Internet drama.

What is a paywall?

A paywall is a restriction put in place by the publishers of journalistic content to ensure only paying members can access the page.

There are multiple types of paywalls:

  • metered paywall: Only takes effect after a certain number of articles have been read. Can usually be circumvented by clearing the browser’s session data. Unless the site does some advanced fingerprinting – which, if client-side, could itself be blocked – or keys off the IP address, you will look like a brand-new visitor.
  • dynamic paywall: Only takes effect after an article proves to be popular or some other criterion kicks in. Thus, if The Archive captured the page sufficiently early, there is nothing to bypass.
  • soft paywall: Only affects a subset of all articles; the rest can be read by anyone.
  • client-side paywall: The entire content is transferred to the user’s browser and hidden with CSS or JavaScript.
  • hard paywall: Affects all content of the site and all users without a subscription, in particular all anonymous (i.e. not logged in) users.

For now, we are only interested in hard paywalls, like the ones the Wall Street Journal or the Financial Times have in place. This type of paywall is the most challenging.

How to use The Archive

It’s pretty simple. You go to archive.today, paste the URL of the article you want to read, and you are usually presented with the full contents of the article. If the site hasn’t been archived yet, you will be offered the option to archive the page. It will take a couple of moments, and then you will be redirected to the full article, like this:

An article from the WSJ on archive.today with the paywall bypassed

For comparison, this is the original:

An article from the WSJ on wsj.com with the paywall in place

So how does it do that? There are several hypotheses that people have put forward.

Hypothesis 1: Origin spoofing

The first idea is to pretend to be someone who is granted access without authenticating. There are certain visitors who are privileged, like search engines, because publishers want their content to be indexed by Google and other search engines.

Sometimes, publishers may want to allow visitors coming from Google to access the content, but not the visitors who receive the link to an article from someone who found it on Google. Other times, they may allow at least a certain number of visitors from aggregators like Reddit, Twitter, and so on, so their articles have a chance to go viral.

One way to fake this is to set the referer header [sic] to one of the allowed sites. Request headers are entirely controlled by the client, and there is no obligation for a client to be truthful. This way, you can easily pretend that you are a visitor sent from a friendly site.

Pretending to be a Google crawler bot is a bit trickier. Sure, you could try to set the User-Agent header to Googlebot or some other value that they use, but it’s not that simple. Google has published details about their crawler infrastructure, including how to correctly verify requests from Google:

Google’s crawlers identify themselves in three ways:

  1. The HTTP user-agent request header.
  2. The source IP address of the request.
  3. The reverse DNS hostname of the source IP.

The third point is crucial. The reverse DNS hostname cannot be faked, because Google controls the relevant DNS records. So unless all the biggest news outlets – who have a great interest in blocking paywall bypasses – are careless, impersonating a search engine isn’t feasible and can’t explain the mystery.

One caveat, though: being careless is not far-fetched. Determining the source IP address can be tricky. The application that decides whether to deliver the full content or just the teaser could (and probably does) sit behind one or even several reverse proxies. So naively looking at the underlying connection would always yield the proxy’s IP address, which is pretty useless. That’s why the proxy can attach the Forwarded header to preserve the true source IP address. Except the client can set that header, too. And if the proxy does not remove it, and the application reads the wrong header, it could happen that the application interprets the client-set header as the true one.

The fact that there are similar headers like X-Forwarded-For, X-Forwarded-Host, and X-Forwarded-Proto makes this even trickier. Depending on various libraries and software components used in the entire stack, this can easily go wrong.

But we can conclude that if you are sufficiently careful, origin spoofing is not the explanation.

Hypothesis 2: AMP and privileged proxies

The Archive’s operator was once asked directly how they bypass paywalls. They replied:

Often the AMP version has more free content than the regular web page. For these sites, an AMP version is downloaded even if a regular version is requested (by replacing www. with amp. or adding ?amp=1 to the end, etc) It sacrifices accuracy, but it gives people what they expect. Some browser extensions do the same thing. It can also be done manually.

As they concede with the first word, this isn’t the full explanation. But it’s true that it can work in some cases. AMP (Accelerated Mobile Pages) is a technology that was developed by Google to optimize the browsing experience on mobile devices, and if the publishers aren’t careful, they might leak content in that alternative format. Google also caches AMP pages in their Content Delivery Network (CDN), so we might find the content we’re looking for there, but only in that case.

Another trick is to use Google as a privileged visitor and get them to forward what they see to us. One example is Rich Results Test, meant for web developers to test how their page looks to a Google crawler bot. However, this doesn’t work for sites like the WSJ, because presumably they correctly follow the steps described by Google to identify crawler bots.

Hypothesis 3: Crowdsourcing

Could it be that some users with subscriptions are just good Samaritans and submit the content they can access to The Archive? I doubt it. When you submit a URL to The Archive, its code running in your browser cannot access third-party sites in your session, due to the browser’s same-origin policy.

Imagine if it could: for example, it could read highly sensitive data like your banking statements as soon as you visit their site, without you knowing.

Is it possible that it uses a browser add-on? Technically, yes, but there are only third-party add-ons (I believe there used to be at least a blessed one, but it’s not linked anymore), over which the owner has no control. In any case, I once reviewed one and it simply used the API of The Archive. No data other than the target URL was transmitted from the user’s browser.

Hypothesis 4: They’re simply paying up

If the publisher does everything right, there is only one way to get access: by buying a subscription. It’s entirely possible that The Archive is simply paying up. This claim is hard to verify, but we can discuss plausibility.

Looking at the screenshot above from the WSJ, we can see the “Sign In” button, which would probably be replaced by something like a “My Profile” button if the visitor were logged in. This would suggest that The Archive does not pay, but it’s also quite feasible to only replace the teaser with the full article and leave the header and everything else as is. It would probably require careful manipulation that wouldn’t scale easily to hundreds of different news outlets, but it’s doable.

I don’t know if publishers embed a watermark in their articles specific to the subscribing account to catch these sorts of shenanigans, but if they do, it would probably be advisable to remove that watermark along with the “My Profile” button – meaning some modification would be necessary anyway.

It’s reasonable to assume that when publishers suspect that an account is sharing content on a large scale, they will terminate the subscription. This could explain why the bypass would sometimes fail. I don’t have data supporting this claim, but if The Archive can sometimes bypass the WSJ paywall and sometimes it can’t, this would fit. The operator needs time to buy a new subscription.

Client-side solutions

It would be nice if there were a browser add-on that would provide the same service as The Archive. I wouldn’t have to rely on some dubious service (which also might disappear eventually) and it would be more convenient. And indeed, there are such add-ons, most notably “Bypass Paywalls Clean” by someone who calls themself Magnolia1234. They don’t publish the source, but you can inspect the published add-on and you’ll find that as a last resort, they delegate the functionality to The Archive.

Similarly, there is Wallhop.io, which claims to specifically bypass paywalls. However, looking at their blog, we learn that they first employ various techniques (probably similar to those described above), and then:

If every method fails, fall back to an archived copy from archive.is.

I am unaware of any add-on – or even another service – that is as powerful as The Archive in bypassing paywalls without eventually resorting to The Archive itself.

Conclusion

Publishers face the challenge of blocking full access to their content for unauthenticated visitors except for a small, fuzzy, hard-to-recognize subset. Paywall bypasses exploit small oversights or inaccuracies in the developers' detection mechanisms – oversights they either don’t know about or can’t be bothered to fix. There is no one trick to bypass all paywalls, so the bypass tools use a combination of several techniques. As a last resort, they delegate the bypass to The Archive, whose operator is probably simply paying for subscriptions.