TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Show HN: Jam API, turn any site into a JSON api using CSS selectors

300 点作者 gavino大约 9 年前

26 条评论

lost_my_pwd大约 9 年前
A little word of warning&#x2F;encouragement. I did something similar a long time ago (JSONDuit), which got posted to HN by someone else.<p>You will probably run into a healthy mix of &quot;that&#x27;s cool&quot; &#x2F; &quot;I did that before you!&quot; &#x2F; &quot;but how will it make money?&quot;. Ignore it and do your thing. If you figure out how to monetize it, great! Even if you don&#x27;t or if you have no desire to, you will have learned and grown during the course of the project. That is invaluable.<p>Have fun and screw the haters...
评论 #11588324 未加载
评论 #11588609 未加载
评论 #11589645 未加载
评论 #11588360 未加载
adriancooney大约 9 年前
This is a fantastic idea and I&#x27;m really surprised nothing like this has existed before, it seems like such a no-brainer. Great work.
评论 #11587815 未加载
评论 #11587994 未加载
评论 #11588555 未加载
评论 #11588123 未加载
评论 #11587670 未加载
评论 #11588154 未加载
评论 #11587882 未加载
评论 #11590280 未加载
ptwt大约 9 年前
I put this similar project[0] together a while ago. Almost the same concept, but I skipped the json layer altogether as I just wanted a quick way of getting nuggets of content from webpages into my terminal.<p>For example:<p><pre><code> curl https:&#x2F;&#x2F;news.ycombinator.com&#x2F;news | tq -tj &quot;.title a&quot; </code></pre> 0. <a href="https:&#x2F;&#x2F;github.com&#x2F;plainas&#x2F;tq" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;plainas&#x2F;tq</a>
评论 #11592500 未加载
jstanley大约 9 年前
with curl:<p><pre><code> $ curl -d url=https:&#x2F;&#x2F;news.ycombinator.com&#x2F; -d json_data=&#x27;{&quot;title&quot;:&quot;title&quot;}&#x27; http:&#x2F;&#x2F;www.jamapi.xyz&#x2F; </code></pre> =&gt;<p><pre><code> { &quot;title&quot;: &quot;Hacker News&quot; } </code></pre> Also, the Ruby example appears to post to the wrong URL?
评论 #11587568 未加载
chriswarbo大约 9 年前
Very nice idea. Although scraping should always be a last resort, I could imagine using this for semi-serious purposes, i.e. when I care enough about the output, will be doing many requests, don&#x27;t mind relaying data via a third-party, etc.<p>I currently do quite a bit of scraping for my own use (generating RSS feeds for sites, making simple commandline interfaces to automate common tasks, etc.). I&#x27;ve found xidel to be pretty good for this: it starts off pretty simple (e.g. with CSS selectors or XPath), but gets pretty gnarly for semi-complicated things. For example, it allows templating the output, using a language I struggle to grasp. This service seems to address that middle ground, e.g. restricting its output to JSON, and hence making the specification of the output much simpler (a nice JSON structure, rather than messing around with splicing text together).
NicoJuicy大约 9 年前
I&#x27;m actually wondering if it would be possible to add forms authentication to this?<p>Eg. Post with some sort of css selecters and then a &quot;cookie memory&quot;.
评论 #11592613 未加载
fryiee大约 9 年前
Great! I&#x27;ve been trying to get my head around Scrapy, and I have little Python experience. This seems to fit in a lot better with my skillset for the project I&#x27;m working on.
denishaskin大约 9 年前
Application Error An error occurred in the application and your page could not be served. Please try again in a few moments.<p>If you are the application owner, check your logs for details.
OJFord大约 9 年前
Yes, yes, yes!<p>I&#x27;m using Apifier at the moment, which I really like, but my biggest gripe is the awkwardness of source (and VCS) integration. The best I&#x27;ve come up with is to export the JSON config (which contains the scraper source code as a value - yuck) and try to remember to keep re-exporting and checking it in.<p>Having also had to hack around the inability to parameterise the scrape url (e.g. &#x27;profile&#x2F;$username&#x27;) - which they&#x27;ve since added support for - I started to wonder if I mightn&#x27;t as well just use BeautifulSoup (Python HTML parser lib) and check it in properly.<p>This is probably my ideal. I can keep it all in source control because it&#x27;s just an HTTP request body, and I can parameterise it because, well, it&#x27;s just an HTTP request body!<p>It&#x27;s also open source because you&#x27;re an amazing person; so if I had one little concern left about the availability of your site I can dismiss it right away since I could run my own on Heroku should jamapi.xyz prove unsustainable. It&#x27;s possibly a better idea to do that anyway, but I often wonder if Heroku doesn&#x27;t consider that a problem - multiple instances of the same app running on free dynos under different accounts...
staticelf大约 9 年前
I just get &quot;invalid json&quot; when I try to use the form on the page.
soheil大约 9 年前
I think with advent of tools like this developers more and more will be thinking of ways to make it hard to have someone scrape their website into data structures. I wonder if we are going to see the same thing that happened to minimized js happening to html more and more. I know there are sites that dynamically change css class names and ids. But I think soon we will also see div hierarchies to dynamically change form without presentationally looking different to the end user.
评论 #11591667 未加载
WA大约 9 年前
HTTPS results in 500 Internal Server Error.<p>Edit: Well no, it&#x27;s only some sites. E. g. <a href="https:&#x2F;&#x2F;medium.com" rel="nofollow">https:&#x2F;&#x2F;medium.com</a>
评论 #11587681 未加载
评论 #11587667 未加载
评论 #11587659 未加载
MetaMetaApplyHN大约 9 年前
Does anyone have any information on anyone that&#x27;s used HTTP as an API to share&#x2F;create metadata for any transactions, content, etc. publicly online? I would very curious to know about it!<p>Welcome feedback on my &quot;Apply HN&quot; on doing exactly this: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11583348" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=11583348</a>
评论 #11588877 未加载
loisaidasam大约 9 年前
Might be helpful to have the example execute inline so you can see what&#x27;s going on&#x2F;experiment without having to leave the page.
splatcollision大约 9 年前
Nice work, thanks for adding the Github link. I can think of lots of immediate use for this. Consider publishing on NPM?
bartkappenburg大约 9 年前
OT perhaps: I&#x27;m still looking for a solution that has a graphical UI that allows users to point and click an element on their page and returns the corresponding CSS-selector. SelectorGadget does this as a chrome-extension but I&#x27;m looking for something that works without an extension.
评论 #11587935 未加载
评论 #11587938 未加载
评论 #11588263 未加载
daw___大约 9 年前
Wonderful idea.<p>What about DOM nodes generated by JavaScript? Will Jam render the page before scraping?
评论 #11587849 未加载
karlcoelho1大约 9 年前
If anyone remembers, they was a YC company that did exactly this. It was called Kimono Labs. I think it failed and just got acquired a year ago. &quot;Jam API&quot; will probably do way better because, well, open source.
paulmd大约 9 年前
I&#x27;ve been thinking about writing some website-to-JSON scrapers myself and this basically solves that problem (since I would have been going after CSS selectors or xpath anyway myself). Nice job.
dimino大约 9 年前
How will someone like CloudFlare stop a tool like this from scraping their customer&#x27;s sites? Just blocking the tool&#x27;s IP?
评论 #11592322 未加载
smadge大约 9 年前
I wish site publishers annotated their markup with RDFa tags so every web page was already an &quot;api&quot;
nsgi大约 9 年前
If it&#x27;s going to be used for serious purposes it really needs HTTPS support, as most APIs do these days.
thomasahle大约 9 年前
What do you think would be a good syntax for enabling following links?<p>Say I wanted the Hacker News links + first comment?
评论 #11587881 未加载
uberneo大约 9 年前
<a href="http:&#x2F;&#x2F;blog.webkid.io&#x2F;nodejs-scraping-libraries&#x2F;" rel="nofollow">http:&#x2F;&#x2F;blog.webkid.io&#x2F;nodejs-scraping-libraries&#x2F;</a> -- Good scraping options in NodeJS .. my personal favourite is <a href="https:&#x2F;&#x2F;github.com&#x2F;rc0x03&#x2F;node-osmosis" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rc0x03&#x2F;node-osmosis</a>
amelius大约 9 年前
Isn&#x27;t this exactly what XML (or for that matter XHTML) was supposed to do?
评论 #11593202 未加载
joelbondurant大约 9 年前
This!... is why we can&#x27;t have nice things.