Archivists work to save disappearing data.gov datasets

I&#x27;m quoted in this article. Happy to discuss what we&#x27;re working on at the Library Innovation Lab if anyone has questions.<p>There&#x27;s lots of people making copies of things right now, which is great -- Lots Of Copies Keeps Stuff Safe. It&#x27;s your data, why not have a copy?<p>One thing I think we can contribute here as an institution is timestamping and provenance. Our copy of data.gov is made with <a href="https:&#x2F;&#x2F;github.com&#x2F;harvard-lil&#x2F;bag-nabit">https:&#x2F;&#x2F;github.com&#x2F;harvard-lil&#x2F;bag-nabit</a> , which extends BagIt format to sign archives with email&#x2F;domain&#x2F;document certificates. That way (once we have a public endpoint) you can make your own copy with rclone, pass it around, but still verify it hasn&#x27;t been modified since we made it.<p>Some open questions we&#x27;d love help on --<p>* One is that it&#x27;s hard to tell what&#x27;s disappearing and what&#x27;s just moving. If you do a raw comparison of snapshots, there&#x27;s things like 2011-glass-buttes-exploration-and-drilling-535cf being replaced by 2011-glass-buttes-exploration-and-drilling-236cf, but it&#x27;s still exactly the same data; it&#x27;s a rename rather than a delete and add. We need some data munging to work out what&#x27;s actually changing.<p>* Another is how to find the most valuable things to preserve that <i>aren&#x27;t</i> directly linked from the catalog. If a data.gov entry links to a csv, we have it. If it links to an html landing page, we have the landing page. It would be great to do some analysis to figure out the most valuable stuff behind the landing pages.

One of the USA greatest strengths is the almost unprecedented degree of transparency of governments records going back decades. We can actually see the true facts including when our government has lied to us or covered things up. Many other nations do not have this luxury and it has provided the evidentiary basis for both legal cases and &quot;progress&quot; in general. Not surprising that authoritarians would target and destroy data as it makes their objective of a post-truth society that much easier

Beyond federal websites (.gov, .mil) there are lot of gov contractor websites that are being taken down (presumably at the demand of agencies) that contain a wealth of information and years of project research.<p>Some below of contractors that work with US AID:<p>- <a href="https:&#x2F;&#x2F;www.edu-links.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.edu-links.org&#x2F;</a> (taken down)<p>- <a href="https:&#x2F;&#x2F;www.genderlinks.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.genderlinks.org&#x2F;</a> (taken down)<p>- <a href="https:&#x2F;&#x2F;usaidlearninglab.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;usaidlearninglab.org&#x2F;</a> (taken down)<p>- <a href="https:&#x2F;&#x2F;agrilinks.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;agrilinks.org&#x2F;</a> (presumably at risk)<p>- <a href="https:&#x2F;&#x2F;www.climatelinks.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.climatelinks.org&#x2F;</a> (presumably at risk)<p>- <a href="https:&#x2F;&#x2F;biodiversitylinks.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;biodiversitylinks.org&#x2F;</a> (presumably at risk)

I’ve been archiving data.gov for over a year now and it’s not unusual to see large fluctuations on the order of hundreds or thousands of datasets. I’ve never bothered trying to figure out what exactly is changing, maybe I should build a tool for that…

&gt; The outlet reports that deleted datasets &quot;disproportionately&quot; come from environmental science agencies like the Department of Energy, National Oceanic and Atmospheric Administration (NOAA), and the Environmental Protection Agency (EPA).<p>Was there an EO targeting these areas?

Related ongoing thread:<p><i>CDC data are disappearing</i> - <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42897696">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42897696</a> - Feb 2025 (216 comments)

What&#x27;s a good way to be an &quot;Archivist&quot; on a low budget these days?<p>Say you have a few TBs of disk space, and you&#x27;re willing to capture some public datasets (or parts of them) that interest you, and publish them in a friendly jurisdiction - keyed by their MD5&#x2F;SHA1 - or make them available upon request. I.e. be part of a large open-source storage network, but only for objects&#x2F;datasets you&#x27;re willing to store (so there are no illegal shenanigans).<p>Is this a use case for Torrents? What&#x27;s the most suitable architecture available today for this?

Still, even with best efforts this is such a shame. There is always going to be a question around governance over the data, integrity, and potentially chain of custody as well. If the goal is to muddy the waters and create a narrative that whatever might be in this data isn&#x27;t reliable or accurate then mission accomplished. I don&#x27;t see how anything can stop that.<p>Not to say the data isn&#x27;t incredibly valuable and should be preserved for many other reasons of course. All the best to anyone archiving this, this is important work.

Related ongoing discussion<p>The government information crisis is bigger than you think it is - <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42895331">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=42895331</a>

Is this normal when there&#x27;s change in presidency?

Are datasets mirrored anywhere where the govt doesn&#x27;t automatically have a take down authority? If not there should be a mirroring effort.

I don’t see a list of the datasets that have gone missing. Is there a list?

Does anyone know if the St Louis Federal Reserve (and I guess the federal reserve banks generally) is subject to presidential executive orders or is it entirely responsible to the Federal Reserve Board and the St. Louis Bank president? FRED is the only dataset I access regularly

Do we know what datasets these are? Do we actually have a diff here so we know what&#x27;s been removed? There&#x27;s a lot of assumptions being thrown around here, but we don&#x27;t even know if this is some kind of malicious compliance. An actual list of what&#x27;s been removed would probably clear the air a lot.<p>As one of the reddit comments (in the thread linked by the article) pointed out,<p>&gt; During the start of Biden’s term, On 6th feb data.gov had “218,384 DATASETS” but on 7th feb it only had “192,180 DATASETS”

It&#x27;s impressive that volunteers are stepping up to archive this. I understand the desire to keep this open data available.<p>How much of this sort of effort results in that data being used? Are there success stories for these datasets being discoverable enough and useful to others?

If the intention is to restore these data sets at some future date, when sanity has possibly been restored, then there needs to be a way to demonstrate that the archived data hasn&#x27;t itself been modified. Without that, malign actors (e.g. oil&#x2F;gas lobby) could very easily poison the future.

I think people are interested in archiving and the political image associated with that but I don’t think anybody cares about the content. Who is going to go back and read Biden era agency publications?

Already seeing: 404 Not Found: Requested route (&#x27;ed-public-download.app.cloud.gov&#x27;) does not exist.

don&#x27;t they have to have to have done this &#x2F;before&#x2F; it gets deleted?

I hope volunteers and others are able to save as much as possible of the data.<p>Removing and altering of the information and data is one of the fundamental threats in our digital world.<p>It probably makes the most sense to do this on a daily basis. If something new, if published, grab it as soon as possible.<p>Data can also be redacted or altered for a variety of reasons, being able to see the before and after states can be illuminating.<p>Something I feel is missing here are statistics for each administration.<p>Does this only happen under a Trump administration, or does it happen to smaller or larger extent under other administrations?<p>I don&#x27;t know how far back this federal goes so it might not be easy,

Tbh, im kind of surprised these things weren&#x27;t being archived as they were being published. Trump is an extreme case, but its not the first time a change in administration resulted in removing old websites.

I read, in past days, that the man who ordered the construction of the nearly infinite Wall of China was that First Emperor, Shih Huang Ti, who likewise ordered the burning of all the books before him. That the two gigantic operations - the five or six hundred leagues of stone to oppose the barbarians, the rigorous abolition of history, that is of the past - issued from one person and were in a certain sense his attributes, inexplicably satisfied me and, at the same time, disturbed me.<p>- Borges

Let&#x27;s make torrents and seed them.

First week we had mass deportation, second week we&#x27;ve heard of the building of concentration camps for undesirables, and now the modern version of book burning. There&#x27;s something different about this republican government.

99.9% of commenters here seem to have missed this:<p>&gt; For example, in the days after Joe Biden was inaugurated, data.gov showed about 1,000 datasets being deleted as compared to a day before his inauguration<p>It&#x27;s almost like this stuff happens regularly. If &lt;insert Dem savior&gt; wins in 2028, tons of government websites will also change in the first couple weeks of their presidency. Is it because they&#x27;re a fascist dictator? Or is it because those websites reflect the administration&#x27;s viewpoints on issues?<p>Wish people would take a deep breath and step back and think a little more. I despise Trump, but there&#x27;s crying wolf and then there&#x27;s the current state of media and online discourse. Trump thrives in this type of environment. He purposefully fosters it. Playing gotcha with him doesn&#x27;t work because he doesn&#x27;t care.