It is such an awesome library. Totally rocks.<p>To get a quick glimpse of how easy it is to work with look at this screencast:
<a href="http://railscasts.com/episodes/191-mechanize" rel="nofollow">http://railscasts.com/episodes/191-mechanize</a>
Unfortunately, Perl-WWW::Mechanize does not yet handle Javascript, has anybody (Perl, Python, Ruby) a solution for that? DOM+Javascript simulation comes to my mind. Or maybe a fully controllable browser component (where DOM and JS are built-in) with an API. Anyone with first-hand experience?
Also worth checking out is Scrapy: <a href="http://scrapy.org/" rel="nofollow">http://scrapy.org/</a><p>It's a Python scraping framework based on Twisted. It lets you use some of Mechanize's API to describe the scraping jobs.
We use the mechanize ruby gem for integration with other sites and web applications - doing things that are not exposed through APIs but only interface. Really cool!<p>Way better than Net::HTTP