For nearly three decades, the Wayback Machine has been the internet\u2019s safety net. When a government agency quietly deleted a policy document, a journalist could find the original on archive.org. When a company changed its terms of service without telling anyone, the old version was still there. When a news article was edited after publication to remove inconvenient facts, the original was preserved.<\/p>\n
The Wayback Machine archived over one trillion web pages. Researchers, journalists, historians, lawyers, students, and everyday internet users have relied on it as the definitive backup of the public web.<\/p>\n
In 2025 and 2026, that safety net developed serious holes.<\/p>\n
Major publishers, including The New York Times, The Guardian, Reddit, and more than 340 local news outlets across nine countries, have moved to block the Wayback Machine\u2019s crawlers from accessing their content. Page captures among news publications dropped 87 percent between May and October 2025 alone. The tool that everyone assumed was preserving the web is now being actively prevented from doing so by the very organizations whose content most needs preserving.<\/p>\n
This guide explains exactly what happened, why the Wayback Machine cannot be the sole solution anymore, and what practical steps you can take right now to build your own reliable web archive.<\/p>\n
The Wayback Machine did not change. The web around it did.<\/p>\n
As AI companies raced to build and train large language models, they needed enormous quantities of high-quality text. Archived news content<\/a> is exactly that: structured, dated, attributed, professionally written text accumulated over decades. The Internet Archive\u2019s massive repository of over one trillion archived pages made it an attractive source for AI training pipelines.<\/p>\n A 2023 analysis by The Washington Post found that data from the Internet Archive had appeared in major AI training datasets. The domain for the Wayback Machine<\/a> was among the top 200 most represented sources in the C4 dataset used to train Google\u2019s T5 model and Meta\u2019s Llama models. Publishers who were already filing copyright lawsuits against AI companies saw the Wayback Machine as a gap in their defences. The result: major publishers began blocking the Wayback Machine\u2019s crawlers from accessing their content, not because they oppose archiving in principle, but because they could not distinguish between the Archive\u2019s preservation mission and AI companies using the Archive as a training data pipeline.<\/p>\n The New York Times spokesperson stated directly: \u201cThe issue is that Times content on the Internet Archive is being used by AI companies in violation of copyright law to directly compete with us.\u201d The Times implemented what the Wayback Machine\u2019s own director described as a \u201chard block\u201d, going beyond standard robots.txt conventions to actively prevent archival access.<\/p>\n Reddit announced in August 2025 that it would block Wayback\u2019s crawlers. USA Today Co.\u2019s decision to block access effectively removed hundreds of local newspapers from the historical record simultaneously. The Guardian restricts access to its journalism. The Financial Times blocks broadly.<\/p>\n The Wayback Machine\u2019s director, Mark Graham, described the situation clearly: \u201cWe are collateral damage.\u201d<\/p>\n The stakes are bigger than one tool.<\/b><\/p>\n Wikipedia links to over 2.6 million news articles preserved by the Wayback Machine across 249 languages. Courts have used archived pages as evidence. Journalists have used them to prove government agencies changed official statements. When publishers block archival access, they are not just limiting one tool; they are removing those pages from the historical record entirely. As EFF put it: \u201cSacrificing the public record to fight those battles would be a profound, and possibly irreversible, mistake.\u201d<\/p>\n Even before publishers began blocking its crawlers, the Wayback Machine had inherent limitations that many users did not fully understand. The blocking crisis has made these gaps more consequential, but they existed before it.<\/p>\n The Wayback Machine cannot archive content behind logins, paywalls, or authentication walls. A news article that requires a subscription, a government portal that requires an account, and a research database that requires institutional access, none of these are archived. If the page you need requires any form of login, the Wayback Machine cannot help.<\/p>\n The web of 2026 is very different from the web of 2001 when the Wayback Machine was launched. Modern websites built on React, Vue, Angular, and similar frameworks render their content dynamically through JavaScript execution. When the Wayback Machine\u2019s crawler visits these pages, it often receives an empty HTML shell, the structure of the page without the actual content, which only loads after JavaScript runs. The archived version of many modern pages is therefore incomplete, showing a layout without content.<\/p>\n The Wayback Machine does not archive every page every day. Popular pages may be archived frequently. Obscure pages may be archived once a year or less. If a page you need was only archived six months before it was changed or deleted, the version you find may be significantly out of date. You cannot rely on the Wayback Machine to have captured any specific page at any specific time.<\/p>\n The Wayback Machine has historically respected the robots exclusion standard; if a website tells crawlers to stay out via robots.txt, the Wayback Machine complies. This means any website that has ever added robots.txt exclusions may have gaps or complete absences in its Wayback Machine archive, regardless of the AI blocking situation.<\/p>\n In October 2024, the Internet Archive suffered a major cyberattack that compromised 31 million user accounts and took the Wayback Machine offline for weeks. In November 2025, another disruption took the service temporarily offline. In May 2023, an AI company\u2019s automated requests caused a server overload that took the Archive offline temporarily. A service that depends on a single nonprofit organization operating on a limited budget carries inherent fragility risks.<\/p>\n The fundamental limitation<\/b><\/p>\n The Wayback Machine is a passive archiver. It crawls the web on its own schedule, archives what it can access, and makes those archives available to the public. It does not archive on demand for specific users, cannot access gated content, cannot guarantee any specific page was captured, and cannot be relied upon in 2026 to have archived content from major news publishers. If you need a specific page archived, the only person you can rely on to do it is yourself.<\/p>\n The solution to the Wayback Machine\u2019s limitations is the same solution that has always been available for anyone who needed to be certain a specific page was preserved: archive it yourself. When you convert a page to PDF the moment you find it, you create a permanent archive, completely under your control, immediately available, and independent of any third-party service\u2019s crawl schedule, robots.txt policies, or operational stability.<\/p>\n The fastest, most reliable method for archiving any web page you find is to convert it to PDF immediately using webs2pdf.com. This creates a complete, permanent snapshot of the page exactly as it appears at that moment, with full content, all images, and no dependency on any third-party archive\u2019s crawl schedule.<\/p>\n Why PDF is the right archiving format<\/b><\/p>\n A PDF preserves the visual appearance of a web page, the full text content in searchable and selectable form, the layout and structure that provide context, and any images that appeared on the page. Unlike a screenshot, the text is searchable and can be copied. Unlike a bookmark, it does not break when the original page changes. Unlike the Wayback Machine, it does not depend on a crawl having happened at the right time. A PDF archive you created yourself is the most reliable record of what a web page contained at a specific moment.<\/p>\n The most critical archiving habit is timing. When you find a news article, research paper, government document, or any web page that matters to your work or research, archive it immediately. Not later today. Not when you get to it. Now.<\/p>\n Every hour that passes between when you find a page and when you archive it is an hour in which the page could be updated, corrected, or deleted without notice. The Wayback Machine\u2019s value was that it captured pages you had already decided you no longer needed to capture. That passive safety net is shrinking. Your active archiving habit needs to fill the gap.<\/p>\n For pages you monitor regularly, a competitor\u2019s pricing page, a regulatory agency\u2019s guidance document, a government portal you track, automated scheduled archiving removes the reliance on both memory and the Wayback Machine. Using webs2pdf.com\u2019s API connected to an automation tool like Zapier, Make, or n8n, you can schedule automatic PDF conversions of specific URLs at regular intervals. This creates a rolling archive of how those pages change over time.<\/p>\n Browser extensions that add a \u201cSave as PDF\u201d button to your browser toolbar make the archiving action as fast as possible. When you are reading a page you want to preserve, one click triggers the PDF save rather than requiring you to navigate to a separate tool. For users who archive frequently, this reduces the friction enough that the habit becomes automatic.<\/p>\nBeyond the Blocking: The Wayback Machine\u2019s Technical Limits<\/h2>\n
1. It only crawls what is publicly accessible<\/h3>\n
2. Modern JavaScript-heavy pages archive incompletely<\/h3>\n
3. Crawl frequency is unpredictable<\/h3>\n
4. robots.txt exclusions are respected<\/h3>\n
5. The archive itself has been attacked<\/h3>\n
What to Do Instead: Build Your Own Web Archive<\/h2>\n
Method 1: Convert to PDF immediately using webs2pdf.com<\/h3>\n
\n
Method 2: Archive news articles immediately upon discovery<\/h3>\n
Method 3: Set up scheduled automatic archiving for critical sources<\/h3>\n
Method 4: Use browser extensions for one-click saving<\/h3>\n
Wayback Machine vs Self-Archiving: A Clear Comparison<\/h2>\n