Hey folks,<p>It's not rare for me to see myself scraping the web for both side projects and personal needs - checking whether the prices of the groceries are good or not based on the local supermarket's website; gathering some emails to cold-email about ideas ...<p>It seems to me that just by having the ability to quickly write some code to gather information from a government web page or anything that shows stuff on the web is a <i></i>huge<i></i> power.<p>What about you? Any great web scraping stories? Have you ever got in trouble by doing so? Made an entire company of it?
I just wrote a crawler for a client who needs to check vendor licenses (plumber, electrician, etc). Given a license number, state, and trade, it looks up the appropriate licensing agency in that state and pulls the license info (issue/expiration dates, biographical information, infractions, etc). Luckily most of these are old school sites so they're easy to crawl, but a few have captcha (easily solved with OCR) or are single page apps.
I think some of the most real-world use cases are in the realestate/hotels/coworking-spaces sector. The second space, email gathering (guess what, all the time people have to use <blabla> at <something> dot com to try to not have their emails scraped).