Asking "What datasets are available to me?" is sometimes the wrong question. A better way of going about the problem is asking something more specific like "How can I create a heat-map of U.S poverty?" The reason why the latter is better is that it not only focuses your attention on something do-able but it actually teaches you more about data analysis than just searching for datasets.<p>For example, to solve the question above you are going to be asking yourself the following followup questions:<p>1) Where do I get a map of the U.S?<p>2) How do I make a heat-map?<p>3) How do I feed in my own data into this heat map?<p>4) What colors do I use?<p>5) Can I do this real-time? Do I need a database? What language do I use?<p>6) Whats a FIPS code?<p>7) How do I find a poverty dataset with FIPS codes?<p>8) This poverty dataset doesn't have FIPS codes, but I can join it with this other dataset that does have FIPS codes.
data.gov and other US gov data sites are getting severe cuts even though they're saving money (<a href="http://www.federalnewsradio.com/?nid=35&sid=2327798" rel="nofollow">http://www.federalnewsradio.com/?nid=35&sid=2327798</a>)<p>Very upsetting for fans of open / accessible (government) data.<p>FWIW, petition at <a href="http://sunlightfoundation.com/savethedata/" rel="nofollow">http://sunlightfoundation.com/savethedata/</a>
Hackers & Founders SV is hosting a hackathon[1] in two weeks at the Hacker Dojo in Mountain View. It's going to be geared towards working with Factual's open data API.<p>Factual's[2] goal is to provide an API to connect all those available data sets, and they have a fairly impressive list of data sets available. Factual is very interested in hearing what datasets you want to work with, and they are willing to bust ass to get them available before the hackathon.<p>We still have around 40 RSVP slots open. You can register here: <a href="http://factualhackathon.eventbrite.com/" rel="nofollow">http://factualhackathon.eventbrite.com/</a><p></shameless plug><p>[1] <a href="http://www.hackersandfounders.com/events/16535156/" rel="nofollow">http://www.hackersandfounders.com/events/16535156/</a><p>[2] <a href="http://www.factual.com/" rel="nofollow">http://www.factual.com/</a><p>[3] <a href="http://factualhackathon.eventbrite.com/" rel="nofollow">http://factualhackathon.eventbrite.com/</a>
<a href="http://www.hiv.lanl.gov/content/index" rel="nofollow">http://www.hiv.lanl.gov/content/index</a><p>For sentimental value: HIV sequence data (and other data) from 1980 till now. Did my thesis on these ;-).<p>In general, there is an enormous amount of gene sequence data around, not just HIV.<p><a href="http://www.ncbi.nlm.nih.gov/sites/" rel="nofollow">http://www.ncbi.nlm.nih.gov/sites/</a><p>Whole genome sequences of eukaryotes (including humans):
<a href="http://www.ncbi.nlm.nih.gov/genomes/leuks.cgi" rel="nofollow">http://www.ncbi.nlm.nih.gov/genomes/leuks.cgi</a>
Edit: Whoops, I thought this was an "Ask HN." The below post still stands for anyone who finds it useful.<p>The U.S. Census has an extremely well-documented large data set:<p><a href="http://www2.census.gov/census_2000/datasets/" rel="nofollow">http://www2.census.gov/census_2000/datasets/</a><p>And the documentation is here:<p><a href="http://www.census.gov/prod/cen2000/doc/sf1.pdf" rel="nofollow">http://www.census.gov/prod/cen2000/doc/sf1.pdf</a><p>The software that they provide to go through the data is crappy, however (90's era).<p>I have an equally crappy but more useful to a computer scientist Common Lisp program that will pull out specific fields from the data set based on a list of field names. If you want that, I can dig it up for you.<p>Also, before you start parsing this, it's worthwhile to read the documentation to find out how the files are laid out, and what each field really means. These files are not relational databases, so if you're looking at it through those lenses, confusion will result. In particular, some things are already aggregated within the data set.
<a href="http://aws.amazon.com/publicdatasets/" rel="nofollow">http://aws.amazon.com/publicdatasets/</a> which includes my former advisor's dataset (UF sparse matrix collection) which includes a matrix or two from my research.
I believe Steven Levitt used the Fatality Analysis Reporting System (FARS) from the national highway traffic safety administration (NHTSA) for his seatbelts vs carseats work:<p>ftp://ftp.nhtsa.dot.gov/fars/
At <a href="http://build.kiva.org" rel="nofollow">http://build.kiva.org</a> there are some nice datasets in the "data snapshots" section. I have high hopes we will be releasing a lot more data.
We provide API access to more than 20 million articles (headlines, excerpts). People have done all sorts of interesting things with it - <a href="http://platform.newscred.com" rel="nofollow">http://platform.newscred.com</a>.
You can find many large datasets here, <a href="http://beta.fcc.gov/data/download-fcc-datasets" rel="nofollow">http://beta.fcc.gov/data/download-fcc-datasets</a> , some are over a gigabyte.