Ask HN: Is there a service that offers Common Crawl as an API?

7 点作者 georgehill13 天前

I am trying to do some data analysis work. I don't want the full dataset. I want only two things: give me the hostname, and give me all the pages or URLs with their HTML.

2 条评论

pluto_modadic大约 20 小时前

there's index.commoncrawl.org where you can ask for a domain with wildcards.

phillipseamore13 天前

Not that I know of but there are various tools like <a href="https://github.com/alwalxed/wayurls">https://github.com/alwalxed/wayurls</a>

评论 #43943437 未加载