Hacker News Dataset Update October 2016

&gt; Caution: That command took just over 30 hours to complete on my macbook. (it also killed Finder a couple times and I had to disable spotlight on the folder I was saving all the .json files to)<p>I had a similar job I needed to do a few months ago and used AWS lambda to massively parallelize the work.<p>I was able to bring down what I estimated would take my laptop 30 days down to about an hour by sharding to a ton of small instances.<p>Might be worth a look if you plan on updating this with any regularity.

I noticed the Hacker News dataset that was published to big query was now a year out of date.<p>I have created an updated copy and made it available for download.<p>(This is the last 10MM entries, I can add the rest if people are interested.)

Since the Hacker News API (<a href="https:&#x2F;&#x2F;github.com&#x2F;HackerNews&#x2F;API" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;HackerNews&#x2F;API</a>) used in this scraping is being brought up again, I&#x27;ll ask a burning question: <i>is development of the API dead?</i><p>From the commit notes in that repo, the only changes from the initial release <i>in 2014</i> are &quot;minor README updates.&quot;