Web scraping with Python open knowledge

I recently joined a brilliant web scraping API company called ScrapFly, who provided me with the resources to create lots of open knowledge on web scraping that I always wanted to create!<p>So, to add to this list here are my top 3 favorite article that could expand OP&#x27;s document:<p>0 - central guide to avoiding blocking - this one a tough one because there&#x27;s so much information: request headers, http versions, TLS fingerprinting, javascript fingerprinting etc. I spent almost a month working on these and it was an amazing research experience that I could never afford myself before.<p>1, 2 - xpath and css selector introduction articles where I built a widget into our article that allows to test all css and xpath selectors right there in the learning material.<p>3 - introduction to reverse engineering - quick introduction to using browser devtools for web scraping, how to inspect the network and replicate it in your program. This is where I point all beginners as understanding the browser really helps to understand web scraping!<p>0 - <a href="https:&#x2F;&#x2F;scrapfly.io&#x2F;blog&#x2F;parsing-html-with-css&#x2F;" rel="nofollow">https:&#x2F;&#x2F;scrapfly.io&#x2F;blog&#x2F;parsing-html-with-css&#x2F;</a><p>1 - <a href="https:&#x2F;&#x2F;scrapfly.io&#x2F;blog&#x2F;parsing-html-with-xpath&#x2F;" rel="nofollow">https:&#x2F;&#x2F;scrapfly.io&#x2F;blog&#x2F;parsing-html-with-xpath&#x2F;</a><p>2 - <a href="https:&#x2F;&#x2F;scrapfly.io&#x2F;blog&#x2F;how-to-scrape-without-getting-blocked-tutorial&#x2F;" rel="nofollow">https:&#x2F;&#x2F;scrapfly.io&#x2F;blog&#x2F;how-to-scrape-without-getting-block...</a><p>3 - <a href="https:&#x2F;&#x2F;scrapecrow.com&#x2F;reverse-engineering-intro.html" rel="nofollow">https:&#x2F;&#x2F;scrapecrow.com&#x2F;reverse-engineering-intro.html</a>

A work in progress guide about web scraping in python, anti bot softwares and techniques and so on. Please feel free to share and contribute with your own experience too.

I recently transitioned one of my scraping projects away from selenium to playwright and I must say that the developer experience is way better, in my opinion.<p>I also implemented to receive a telegram message with the debug trace in case of errors in my pipeline, so that I could have the entire scraping flow to analyze. That’s pretty neat.

Plug for <a href="https:&#x2F;&#x2F;commoncrawl.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;commoncrawl.org&#x2F;</a> if you need billions of pages but don&#x27;t want to deal with scraping the web yourself.

If you&#x27;re the kind of person who wants &quot;open data&quot; (read as broadly as you like) and could get it in snapshots direct from the source without having to scrape, what would your ideal format be?<p>I know it&#x27;s a very open ended question.

Good list, confused about the “tabs weighing less” bit. Isn’t that a preference left for the end-devs?<p>Another tip I’ve found is to check if the data is accessible on a mobile app and proxy it to see if there is a JSON API available.

I was reading another thread about webscraping, someone mentioned CSS selectors being way quicker than xpath. I&#x27;m easy either way but apart from a more powerful syntax what other benefits are there?

It can be frustrating learning web scraping with Python when so many sites actively block scraping.

As the second sentence says, it&#x27;s a cat and mouse game, so there&#x27;s no incentive on either side of bot vs anti-bot to share information.

For websites that require auth via Google Auth this is a non starter. There’s no way to bypass its bot detection

Is there an FLOSS project that combines Scrapy scrapers and just makes the results publicly available?

As someone who recently dealt with scraping sites behind cloudflare...I never want to scrape again

I think that if you don’t want to invest a lot of time into learning web scraping and money to get a pool of residential, or even better mobile, proxies it’s easy to quickly get good results with web scraping API like <a href="https:&#x2F;&#x2F;scrapingfish.com" rel="nofollow">https:&#x2F;&#x2F;scrapingfish.com</a> They have good blogposts, for example, for how to scrape public Instagram profiles: <a href="https:&#x2F;&#x2F;scrapingfish.com&#x2F;blog&#x2F;scraping-instagram" rel="nofollow">https:&#x2F;&#x2F;scrapingfish.com&#x2F;blog&#x2F;scraping-instagram</a>