Archiving Democracy: How the U.S. End of Term Web Archive Preserves Government History

Beginning in January 2025, we witnessed an unprecedented removal of material from U.S. federal websites — documents related to education, health, science, and more, gone from public view almost overnight. Dozens of articles appeared on this topic, including this one from IEEE Spectrum, published in February 2025, shortly after the beginning of the new presidential term. The author, Gwendolyn Rak, like many other doing research in this area, noted that not only did individual web pages change or vanish almost overnight, but that the web presence of entire agencies such as the U.S. Agency for International Development (USAID) disappeared from view. Moments like this are exactly why libraries have collaborated to build the End of Term Web Archive.

The Internet Archive has participated in the End of Term (EOT) Archive since 2004, working alongside partner institutions to capture US government websites at every presidential transition. Over time the scope has grown, but the purpose remains the same: to preserve government information and civic memory at critical moments of democratic change.

What the End of Term Archive Captures

Project partners participate in different ways. Some carry out web archiving, while others lead in compiling lists of sites and pages that should be included, and others conduct research on findings. The crawl is carried out in distinct phases, including a pre-election capture, captures during the election, and finally captures pre- and post-inauguration.

Lessons from the 2024/2025 Crawl

When asked to reflect on what the team learned from the 2024/2025 crawl, Mark Graham, Director of the Wayback Machine said, “We always learn that no matter how much planning we always wind up missing material that should have been included and in scope. To do a thorough comprehensive crawl is a daunting task and we have gaps. More funding and access to resources would help us do a better job. Many people expect that this work is done by the government, but it is not. It needs to happen with trust and integrity, and we provide that.”

“Another challenge is that so much government information is now published through commercial platforms, for example X and various Meta properties. This presents both operational and policy issues around archiving these materials. Widespread Akamai and Cloudflare implementation has, I think, unintended consequences for web archiving of all sorts.”

“Here is an example of important materials that are hosted on social media platforms that came into sharp focus after the wrap up phase from the 2024 crawl, when the US State Department announced they were going to remove social medial posts from prior administrations, including accounts of representing embassies and officials.This caused the team to go back and look at that area in particular. We identified and captured information from more than 700 accounts on the X platform alone. While we worked quickly to archive before deletion, some slipped through the cracks. For the first time we are considering a FOIA [Freedom of Information Act] request. If successful, we may use that to fill in the blanks.”

What’s Now Available

With the 2024/2025 crawl complete, petabytes of material are now available for bulk download and playback via the Wayback Machine. New to this crawl: two million videos, captured as part of the End of Term effort for the first time at this scale.

I also asked Mark to share what he is most proud of in this work. “I’m grateful that we have the opportunity to do this, along with participating partners and the support of funders. We have ample experience to carry this work out, and we have gotten better at this over time. We have seen the evidence and value of this work reflected in what has become a national conversation when so much material was removed from public access starting in January 2025. Because we have done the work, we have evidence about the removal of fundamental documents related to education, health, science and the like, and millions of web pages and web sites deleted abruptly. We’ve seen over 1000 news articles cite this work specifically.”

The End of Term Web Archive is Democracy’s Library in action. Working with the right partners, the right infrastructure, and sustained commitment, it is possible to safeguard the public record for the people it belongs to.