TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Scraping Hacker News on a Schedule with TaskPipes

39 pointsby f292over 9 years ago

6 comments

tynmanover 9 years ago
What Open Source projects are similar? It seems like data transformation tasks like this are a universal problem. I'd just do map/reduce, but that's the advantage that comes with being a developer. Great tools for non-developers to do limited programming tasks are incredibly useful.
juanescobarcomover 9 years ago
Check out Diffbot @ <a href="http:&#x2F;&#x2F;www.diffbot.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.diffbot.com&#x2F;</a> -- you can setup repeat crawls and extract data into .csv&#x2F;Excel file format or JSON with Crawlbot API, extract data automatically with automatic APIs, or Custom API Toolkit. You can get a free trial account at <a href="https:&#x2F;&#x2F;www.diffbot.com&#x2F;plans&#x2F;trial" rel="nofollow">https:&#x2F;&#x2F;www.diffbot.com&#x2F;plans&#x2F;trial</a> to try it out, and you can get paid plans at <a href="http:&#x2F;&#x2F;www.diffbot.com&#x2F;pricing&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.diffbot.com&#x2F;pricing&#x2F;</a>.
fulafelover 9 years ago
This general category of tool seems very useful. Apparently there are many Yahoo Pipes followers. <a href="http:&#x2F;&#x2F;www.makeuseof.com&#x2F;tag&#x2F;12-best-yahoo-pipes-alternatives-look&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.makeuseof.com&#x2F;tag&#x2F;12-best-yahoo-pipes-alternative...</a><p>Support for email and HTML tables seem to be some distinguishing features of this one.
评论 #10264948 未加载
chdirover 9 years ago
I&#x27;ve seen something similar more than once on HN but my google-fu is failing. What other similar services are out there that&#x27;ll let you smartly scrape a page into a spreadsheet.<p>In the past, I&#x27;ve tried Google spreadsheets with &quot;ImportXML&quot; option but got frustrated after a bit and resorted to python.
评论 #10265075 未加载
draugadrottenover 9 years ago
Looks pretty but I can&#x27;t find pricing information.
评论 #10264580 未加载
评论 #10264409 未加载
johnongover 9 years ago
I tried NewsBlur as an alternative to Yahoo Pipes. Like it because it has an Android app as well.