I'm working on a large web scraping project and looking to avoid building custom scrapers for every site using something like BeautifulSoup.<p>Anyone have recommendations for up-to-date scraping APIs or platforms that work well with a variety of websites?
I am actually working on a scraping API for serps, currently it only supports Google, may add more search engines if it gets any traction, it isn't ready yet, but hopefully it will be in a couple of days, if you want to give it a go let me know and I can give you a subscription in exchange for some feedback.
If you're looking to scrape any of the major search engines, eBay, Yelp, and a few others, then SerpApi has a massive number of up-to-date APIs to choose from:<p><a href="https://serpapi.com/search-api" rel="nofollow">https://serpapi.com/search-api</a><p>Disclaimer: I work there!
<i>* Shameless plug *</i> Our feed builder combines a web scraper, AI-powered content auto-detection and data transformer to generate custom JSON and RSS feeds for any public webpage.<p><a href="https://newsloth.com" rel="nofollow">https://newsloth.com</a>