This breaks with using standard web scraping methods (non-headless JS engines). Had to deal with this issue recently due to everything being a damn SPA now. Look into Selenium for running headless browsers if looking to scrape the modern web.
I know some people think all scraping is bad or malicious. I'd like to point out this is a perfectly legitimate use case for it, in fact this is how Google Search operates.<p>Web scraping done correctly should be barely noticeable if at all to the operators. Don't send 10,000 req/s, have aggressive delays, make your retries extremely generous, try to avoid pages or actions you know are "heavy". You don't need to update data from every product page every 5 minutes.
I wrote a similar Python library to do Beautiful Soup scraping with basic PageRank and a Flask web app: <a href="https://github.com/argosopentech/argos-search" rel="nofollow">https://github.com/argosopentech/argos-search</a>