Very nice. Worth keeping in mind prior examples for comparison's sake. My favorites so far:<p>- <a href="https://www.opendatanetwork.com" rel="nofollow">https://www.opendatanetwork.com</a> -- what I would call the "Google, for Socrata datasets"<p>- <a href="https://public.enigma.com/" rel="nofollow">https://public.enigma.com/</a> -- One of the best collections of U.S. federal data, with good taxonomy and lots of useful options for refining a search, such as filtering by dataset size.<p>- <a href="https://www.data.gov/" rel="nofollow">https://www.data.gov/</a> -- Not as useful as what most people would want -- e.g. unlike Enigma and Socrata, it's a directory of self-submitted (by the government) data sources, not one in which the data is stored/provided in a standardized way. But it's a pretty good listing, though not sure if it's much better than just using Google.<p>- <a href="https://data.gov.uk/" rel="nofollow">https://data.gov.uk/</a> -- Better than the U.S. version in terms of usability and taxonomy.
Nice. I'm glad Google is making it easier to find public data sets. I wish that these could be filtered by format, so that you could narrow them to CSV, XML, JSON, KML, etc.<p>Another nice resource that I've used in the past is 'toddmotto/public-apis' on Github [0].<p>In the end I would prefer all public data sets to be available over the DAT protocol [1] instead of being hosted only on government or organization websites. A lot of climate data previously made available by the EPA was taken down, and only saved by efforts of volunteers.[2]<p>[0] <a href="https://github.com/toddmotto/public-apis" rel="nofollow">https://github.com/toddmotto/public-apis</a><p>[1] <a href="https://datproject.org/" rel="nofollow">https://datproject.org/</a><p>[2] <a href="https://www.wired.com/2017/01/rogue-scientists-race-save-climate-data-trump/" rel="nofollow">https://www.wired.com/2017/01/rogue-scientists-race-save-cli...</a>
The state of data sharing seems to be still quite sad.<p>* Hosting problems. The first link I tried was already broken.<p>* Format problems. Also the presented data is in all kinds of formats, some "data sets" even require me to read data off images: <a href="https://www.ceicdata.com/en/indicator/germany/gdp-per-capita" rel="nofollow">https://www.ceicdata.com/en/indicator/germany/gdp-per-capita</a>
And even if it's JSON, this is not particularly great either (Unicode support? Large (64bit) integers?).<p>* Update problems. Many data-sets change over time (e.g. GDP). How can I subscribe to updates? "git pull" would be nice.<p>* Provenance problems. I want to know who put which record into the dataset, when and why? "git log" would be nice.<p>* Presentation problems. (This is OK sometimes) I necessarily want to download 5Gb file before I looked into it. The first few rows of the dataset should be presented on the page, with information about it.<p>Wrote down a few more thoughts a while ago here: <a href="https://github.com/HeinrichHartmann/data-sharing#in-the-ideal-world" rel="nofollow">https://github.com/HeinrichHartmann/data-sharing#in-the-idea...</a><p>Approaches I have seen so far in the wild:<p>* figshare.com -- Addresses Hosting and Presentation.<p>* <a href="https://quiltdata.com/" rel="nofollow">https://quiltdata.com/</a> -- (!) looks great. Still exploring.<p>* github.com -- works fine for small datasets (<1GB)<p>* packaging (yum, pkg, pip) -- (?) Not sure if that works, but at least they solve: Hosting, Update, Provenance.<p>This seems to be a wide open problem to me.
Interestingly enough, I built and released something very similar [1] about a month ago using a Google Custom Search Engine.<p>Here is the Show HN for it: <a href="https://news.ycombinator.com/item?id=17789119" rel="nofollow">https://news.ycombinator.com/item?id=17789119</a><p>[1] <a href="https://databasd.com/" rel="nofollow">https://databasd.com/</a>
What is the best way for a website to format data to the public?<p>I already have my presentation, but I can also provide it as a .xls, .csv, sql, or html table.<p>What would be best to help programmers/data scientists use my data?
Nice to see Google trying this again!<p>It's one of those areas they have long attempts at involvement in - e.g. Google Public Data Explorer which never quite reached it's potential, and Freebase which although flawed was good and was shut down after Google acquired it.<p>I like that this is search based! The web is still the best place to publish data - in fact in my view normal Google search is still by far the best way to find datasets, even though it isn't directly designed for that.<p>There's a link from the about page of Google Dataset Search to this help for webmasters on how to mark up content for it - although it is a bit odd, mainly showing how to mark a dataset with a DOI (so good for academics certainly!):<p><a href="https://productforums.google.com/forum/#!topic/webmasters/nPq4BW6iPIA" rel="nofollow">https://productforums.google.com/forum/#!topic/webmasters/nP...</a><p>Just metadata about data feels like a very niche thing to search to me - I'm still not convinced anyone will maintain the metadata well enough to help. Possibly will work in particular domains.<p>Does Dataset Search have some way to search column headings, types or content (of CSV, Excel, JSON etc)? I can imagine a load of operators that would make that really powerful for finding badly meta-marked up datasets deep in the web. Would seem like the obvious extra thing a dataset search would do.<p>Also previews please!!! Just nicely render the fist ten rows of common formats - CSV and Excel to begin with.<p>What part of Google is doing this?
Looks like academic institutional repositories and figshare are doing the heavy lifting here. It's still neat to see Google aggregate everything, but it's not that different from what they do with other services relying on these sources already, and is largely dependent upon how rich these upstream sources are in the first place.
This is nice. I'm working on a similar open source project that is releasing soon called DataLibrary[0]<p>It goes further by bringing this kind of data together into a single API, converting/cleaning into a similar schema where possible.<p>A small write up can be found on github [1]. Any feedback/ideas would be appreciated!<p>[0] <a href="https://www.datalibrary.com" rel="nofollow">https://www.datalibrary.com</a> (not online currently)<p>[1] <a href="https://github.com/reactual/datalibrary/blob/master/README.md" rel="nofollow">https://github.com/reactual/datalibrary/blob/master/README.m...</a>
<a href="https://data.urbanfootprint.com/" rel="nofollow">https://data.urbanfootprint.com/</a> - browse thousands of environmental, social, transportation, and land use datasets.<p>(disclosure - I work there)
Then there's this $500K just awarded by the NSF to build a "Google for data sets". I wonder if, before making these sorts of grants, the NSF looks at what Google and other and other companies are already (or likely) doing.
<a href="https://www.lehigh.edu/engineering/news/faculty/2018/20180820-davison-heflin-jia-dataset-search-engine-nsf-award.html" rel="nofollow">https://www.lehigh.edu/engineering/news/faculty/2018/2018082...</a>
You may want to try [<a href="https://data.opendatasoft.com](https://data.opendatasoft.com)" rel="nofollow">https://data.opendatasoft.com](https://data.opendatasoft.com...</a> -- thousands of datasets available through the same API, usable online, no download required.
Look at <a href="https://knoema.com" rel="nofollow">https://knoema.com</a> which positions itself as a search engine for data with more than 2.5 billion time series available. They provide both visual data discovery through search and navigation as well as API access through Python, R etc.
I also very much like <a href="https://www.figure-eight.com/data-for-everyone/" rel="nofollow">https://www.figure-eight.com/data-for-everyone/</a>. It's not optimized for search but it's an excellent repository of high quality datasets.
Of note is the link below which indicates how you can have your dataset indexed.<p><a href="https://developers.google.com/search/docs/data-types/dataset" rel="nofollow">https://developers.google.com/search/docs/data-types/dataset</a>
Nothing for FRED <a href="https://toolbox.google.com/datasetsearch/search?query=site%3Afred.stlouisfed.org" rel="nofollow">https://toolbox.google.com/datasetsearch/search?query=site%3...</a> hmm...
Using Google to search Google ... <a href="https://toolbox.google.com/datasetsearch/search?query=site%3Agoogle.com" rel="nofollow">https://toolbox.google.com/datasetsearch/search?query=site%3...</a>
Since we have already a nice reference list of open data portals by country I like to add the German portal: <a href="https://www.govdata.de/" rel="nofollow">https://www.govdata.de/</a>
Looks like this doesn’t crawl one of the most important source for data: academic torrents. For example, I searched for<p>ilsvrc lmdb<p>This should have found imagenet data in lmdb format available somewhere but it returned no results.
Maybe I’m missing something but this strikes me as underwhelming. To the point of something that I could do as opposed to something that the firm that created maps and gmail could do.<p>Is it just me?