What to use for JavaScript based websites with strong bot/scraping detection?<p>I find most popular frameworks out there are easily detectable (selenium, puppeteer, etc).<p>I have been using a homebrew'd solution using a native web view that mimics a popular web browser (yet allows me to run arbitrary JS, etc).
Three real paths I'd evaluate<p>0) Tools: Puppeteer and Playwright are the cleanest way I've found to get proper JS page rendering and behavior control. Node is well suited for this but some prefer python.. Since you sound concerned with blocks I'm guessing plain HTML scrapers like BeautifulSoup or Cheerio would be insufficient but they're more robust in terms of sheer volume and overhead of course.<p>1) if you have money the fingerprint avoiding systems like GoLogin are exceptionally good at avoiding detection. But they're not cheap so you would need to have a reasonable budget to use them well. I've had extremely high success with GoLogin myself and if budget wasn't a concern I'd just default to that.<p>2) less expensively you can use headless Chrome (Browserless.io has an excellent docker image for this) and then proxy using 4g/5g proxies. You usually pay by kb so you'd want to be savvy about blocking images etc to manage costs but with a good proxy and decent tuning this also takes you pretty far except some of the more onerous services like Cloudflare and Datadome. There are also v good captcha solving services that seamlessly integrate. I've had very good results with simple proxying this way and well crafted settings in Browserless like making sure stealth plug-ins used and user agent is properly done etc.<p>3) most inexpensive you can simply use Playwright on Browserless (Chrome Headless) and a captcha service with stealth plug-ins and running from a decent quality IP. I'm careful to check things like viewport size, user agent, ghost cursor etc.<p>Data center IPs usually raise flags and even the server should be selected to not have vpn ports or http ports open as those are also antibot signals among others.<p>--<p>I do #3 at a fairly decent volume (<1m page views per day) across a few dozen machines (each with a half dozen IP addresses and separate VLANs for each container) to scrape from some private sites and so long as I stay under a sane rate limit I've had years of success on many sites that are fairly strict about blocking browsers.<p>In all cases I parse the key parts of the page and dump it into mongo for async processing of the data and to allow fixes when sites change. You need to keep an eye on your ETL pipeline and alert when something breaks - I expect once a quarter I have to fix a selector change or something trivial as sites change.<p>This is also a good substack evaluating the various paid options for the toughest sites. <a href="https://substack.thewebscraping.club" rel="nofollow">https://substack.thewebscraping.club</a>
Usually JS based websites operate with some internal API under the hood.<p>Inspect those, and then directly hit those with something like Python or Go.<p>I prefer Python (requests, lxml, BeautifulSoup).<p>Mimic the headers that your browser is using.<p>Have you tried this already and run into issues?
I have used Playwright to write tests for my website. It was <i>so</i> easy and the code is so readable. I can wholeheartedly recommend it if you need to control a browser from your code. I'm now using it to write a crawler for a dozen websites, because anything else would be too tedious.