Since writing a scraper to discover and parse all historical comments/submissions on HN would obviously get me in trouble, would the HN admins be willing to provide a dump of the historical text/metadata from all comments and [local] submissions so I can make a HN Ngram Viewer for the HN public?<p>I work in an academic lab where I'm one of the developers of a system that generates ngram viewers from large corpuses of text, which we call "Bookworms". Here are a few Bookworms we've created:<p>arXiv scientific publications: http://bookworm.culturomics.org/arxiv/<p>US Congress legislation: http://bookworm.culturomics.org/congress/<p>Open Library books: http://bookworm.culturomics.org/OL/<p>Chronicling America historical newspapers: http://bookworm.culturomics.org/ChronAm/<p>Social Science Research Network research paper abstracts: http://bookworm.culturomics.org/ssrn/<p>We have more Bookworms in the pipeline, including historical legislation in the UK and a massive corpus of texts (70MM+ documents) from the National Library of Australia (Trove) spanning multiple centuries. A new GUI for all our Bookworms will also be rolling out shortly. (Preview: http://bookworm.culturomics.org/new_gui_teaser.png).<p>In my opinion, HN be an awesome candidate for an ngram viewer because there are so many subsets of topics that come/go/stay here, such as the frequency of discussions about web technologies, programming languages, companies/services, the NSA, etc.<p>If this is something the HN admins would be interested in, I'd be happy to put it together. If a privacy agreement is desired before passing off any bulk data, that is not a problem as we've gone this route before, albeit only for private ngram viewers we've created for companies, like the NYT, to use internally.
The Octopart team has done a great job with HNSearch, and we really appreciate the huge favor they've done us by providing it. That said, due to limitations on our end around how they integrate with us, they're not able to offer real-time updates or full fidelity ranking snapshots.<p>I'm working on a more comprehensive first-party API for HN, and plan to implement the following, in this order:<p><pre><code> 1) Near-real-time profiles, comments, and stories as JSON.
2) Real-time streaming of profile and item changes.
3) Near-real-time ranking of comments and stories.
4) Real-time streaming of ranking changes.
5) History of ranking changes.
</code></pre>
Sadly, I can't commit to any firm timeline for future progress right now, but know that I'm working on it :)<p>--
Edit: Remove link to broken data file. Fixing it up tomorrow.
You could look into <a href="https://www.hnsearch.com/api" rel="nofollow">https://www.hnsearch.com/api</a> . They provide the search bar functionality on this site.