Might be of interest:
I wrote a post a couple of weeks back regarding our distributed crawling architecture built using perl+redis+gearman<p>How We Built Our 60-Node (Almost) Distributed Web Crawler
<a href="http://hackerne.ws/item?id=4469911" rel="nofollow">http://hackerne.ws/item?id=4469911</a>