Pretty cool analysis! Scrapy is great but if you're looking to extend this further, I recommend checking out the 3taps API. 3taps is the only API I've seen that breaks down the different components of a Craigslist post (e.g. heading, price, number of bedrooms) and makes it available for scraping. Using 3taps will also make it easy for you to extend the analysis to new cities (you literally change a single parameter). And you should set up a cron job (or Heroku scheduler task if a web app) so you don't have to run the scraper manually every day :-)<p>If you want to check out an example of automated Craigslist scraping, you can check out a search tool I built (craigslist-scraper.herokuapp.com) and the accompanying tutorial (baserails.com/apiscraper). Both are best viewed via laptop.