Better Python Scraping - Installing lxml and Beautiful Soup

Ive scraped dozens of sites beautiful.soup is great, but if you want the job done as quickly and cleanly as possible, PyQuery is thr best.<p>Its the same as jquery, but in Python.<p>Working with beatiful soup quickly becomes.long.and messy and tedious<p>With pyquery, you get what you want with just a couple of CSS 3 selectors simplw and nice<p>wow android 2.2 is terrible for inputting text

I'm learning python on the fly, but I tend to ask a lot of question on freenode's #python. Installing lxml wasn't so bad. I just did "pip install lxml" (easy_install lxml should work too) on my Debian VPS and home server. Seemed to work for me.<p>I am sticking with lxml only for my scraping and html5lib to do my richtext parsing.

I much prefer Scrapy (<a href="http://scrapy.org/" rel="nofollow">http://scrapy.org/</a>). BeautifulSoup is pretty outdated.

Why not "sudo apt-get install python-lxml python-beautifulsoup"? Difficult to make the "olde" argument when you're installing dependencies from apt.

Is mechanize (<a href="http://wwwsearch.sourceforge.net/mechanize/" rel="nofollow">http://wwwsearch.sourceforge.net/mechanize/</a>) considered outdated or convoluted? It's what I've used for my scrapings.<p>Also, how well do these other scrapers handle Javascript? I've had to abandon some scrapes from ASP pages because they wouldn't properly handle it.

Maybe I missed it: why aren't you using pip? As I recall, the set up is as simple as: sudo pip install lxml or sudo pip install BeautifulSoup. If you're learning Python, definitely learn pip. Pip will make your life easier! :)

Just to throw in my own "I like X" better. I recently had some pages that Beautiful Soup just choked on and couldn't parse.<p>I like html5lib, which will even spit out a Beautiful Soup parse tree if that's your thing.

Does this page kill the chrome tab for anyone else?