For what its worth, this would be the bash equivalent:<p><pre><code> for i in {1..1000}
do
wget --quiet "somehost/$i" -O "$i.html" &
done
wait
</code></pre>
Takes 1.8 seconds here. But will obviously depend on the network situation and server. And there probably is something faster than wget.
I wonder where the time is going. I suspect that something like atop would show that the CPU is idle for much of the time, which suggests that the problem is not Python or even the GIL.<p>If that's true, there's something ineffective about the way that the requests are being made/handled.<p>AFAIK, the author did not try "make a bunch of requests before waiting for answers". If many of the requests are going to the same server, blasting a bunch of requests down a single socket before waiting for a response should be faster, if only because there's only one ssl handshake, even if the remote server doesn't pipeline request processing. (I suspect that "too many threads" didn't work because that reduced connection reuse.)<p>There's a fairly simple monkey patch to http\httpsconnection that enables request pipelining.<p>It might also be useful to use something like epoll to wait for the sockets.