A Web Crawler with Asyncio Coroutines

Great tutorial, I would love to see this rewritten with the new async / await syntax in python 3.5

One question I have about this - and I might have missed in article is - I&#x27;m all for using asyncio to make HTTP requests. But I see they apparently also use asyncio for &quot;parse_links&quot;. Since parselinks should be CPU op, would it make sense to use fibers to download links and pass them into a thread pool to actually parse them&#x2F;&#x2F;add to queue?<p>I&#x27;m messing around with some of the ParallelUniverse Java fiber implementation and what I do is spam fibers to download pages and send the String response over to another fiber over a channel that maintains a thread pool to parse response body as they come in&#x2F;&#x2F;create new fibers to read these links.<p>I&#x27;m really just doing this to get more familiar with async programming and specifically the paralleluniverse Java libs but one thing I&#x27;m struggling a bit with is how to best make it well behaved (e.g right now there&#x27;s no bound on number of outstanding HTPT requests).

This article is way more important than the web crawler example used to motivate it. It&#x27;s easily the single best thing I&#x27;ve ever read on asyncio, and I&#x27;ve been using it in anger for a year now. I&#x27;ve passed it around my team, and will be recommending it far and wide!

I&#x27;m working on a project that involves lot&#x27;s of web crawling. I&#x27;m not technical at all (I&#x27;m hiring freelancers).<p>While I do have access to great general technology related advice, this post is bound to bring people well versed in crawling.<p>My question is: in terms of crawling speed (and I know this is dependent of several factors) what&#x27;s a decent amount of pages a good crawler could do per day?<p>The crawler I built is doing about 120K pages per day which to our initial needs is not bad at all, but wonder if in the crawling world this is peanuts or a decent chunk of pages?

It would be interesting to compare this Python approach with a Go goroutine approach. The main question is whether Go&#x27;s libraries handle massive numbers of connections well. Since Google wrote Go to be used internally, they probably do.

Node is well-suited for this type of thing and there are numerous libraries to help.

Appreciate the in-depth description. Look forward to working through this in detail.