Web Scraping in 2016

Keep in mind that companies have <i>sued</i> for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: <a href="http:&#x2F;&#x2F;www.informationweek.com&#x2F;software&#x2F;social&#x2F;linkedin-sues-after-scraping-of-user-data&#x2F;d&#x2F;d-id&#x2F;1113362" rel="nofollow">http:&#x2F;&#x2F;www.informationweek.com&#x2F;software&#x2F;social&#x2F;linkedin-sues...</a><p>OKCupid did a DMCA takedown for researchers releasing scraped data: <a href="https:&#x2F;&#x2F;www.engadget.com&#x2F;2016&#x2F;05&#x2F;17&#x2F;publicly-released-okcupid-profiles-taken-down-dmca-claim&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.engadget.com&#x2F;2016&#x2F;05&#x2F;17&#x2F;publicly-released-okcupi...</a><p>Since both of these incidents, I now only scrape if it&#x27;s a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration.<p>(Disclosure: I have developed a Facebook Page Post Scraper [<a href="https:&#x2F;&#x2F;github.com&#x2F;minimaxir&#x2F;facebook-page-post-scraper" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;minimaxir&#x2F;facebook-page-post-scraper</a>] which explicitly follows the permissions set by the Facebook API.)

Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Corporations will cry like babies when their publicly available data (their livelyhood) gets scraped. They will take you to court.<p>They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume personal integrity when you are surfing the internet?<p>I have high hopes that the judicial system some time not too far from now will realize that since the law should be a reflection of the current moral standings it will always be behind, trying to catch up with us and that those who break the law while not breaking the current moral standings are still &quot;good citizens&quot; unworthy of prison or fines.<p>I guess Google won this iteration of the internet because of the double-standars site owners stand by, to allow Google to scrape anything while hindering any competitors from doing the same. There will only be a true competitor to Google when we in the next iteration of the internet realize that searching vast amounts of data (the internet) is a solved problem, that anyone can do as good a job as Google, and move on to the next quirk, around wich there will be competition, and in the end that quirk will be solved, we&#x27;ll have a winner, signaling that is it time to move on to the next iteration.

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I&#x27;ve gotten pretty good a circumventing rate-limiting and most other controls.<p>I suspect I&#x27;m one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt.<p>At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for message passing middleware, SaltStack for automated VM provisioning, and python everywhere for everything else. Using some randomization, and a list of the top n user agents, I can randomly generate about ~800K unique but valid-looking UAs. Selenium+PhantomJS gets you through non-capcha cloudflare. Backing storage is Postgres.<p>Database triggers do row versioning, and I wind up with what is basically a mini internet-archive of my own, with periodic snapshots of a site over time. Additionally, I have a readability-like processing layer that re-writes the page content in hopes of making the resulting layout actually pleasant to read on, with pluggable rulesets that determine page element decomposition.<p>At this point, I have a system that is, as far as I can tell, definitionally a botnet. The only things is I actually pay for the hosts.<p>---<p>Scaling something like this up to high volume is really an interesting challenge. My hosts are physically distributed, and just maintaining the RabbitMQ socket links is hard. I&#x27;ve actually had to do some hacking on the RabbitMQ library to let it handle the various ways I&#x27;ve seen a socket get wedged, and I still have some reliability issues in the SaltStack-DigitalOcean interface where VM creation gets stuck in a infinite loop, leading to me bleeding all my hosts. I also had to implement my own message fragmentation on top of RabbitMQ, because literally no AMQP library I found could <i>reliably</i> handle large (&gt;100K) messages without eventually wedging.<p>There are other fun problems too, like the fact that I have a postgres database that&#x27;s ~700 GB in size, which means you have to spend time considering your DB design and doing query optimization too. I apparently have big data problems in my bedroom (My home servers are in my bedroom closet).<p>---<p>It&#x27;s all on github, FWIW:<p>Manager: <a href="https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;ReadableWebProxy" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;ReadableWebProxy</a><p>Agent and salt scheduler: <a href="https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;AutoTriever" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;AutoTriever</a>

A neat trick I sometimes use to &quot;scrape&quot; data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr:<p><pre><code> $.ajaxSetup({ dataFilter: function (data, type) { if (this.url === &#x27;some url that you want to watch!&#x27;) { &#x2F;&#x2F; Do anything with the data here awesomeMethod(this.data) } return data } }) </code></pre> I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page down every 2 seconds, and the `awesomeMethod` just initiated the download. Pasted it all in dev-tools console, and the cheap &quot;scraper&quot; was ready!

This good list of tactics underscores, for me, how the state of the Web has made it a lot more difficult to teach web scraping as a fun exercise for newbie programmers. It used to be you could get by with an assumption that what you see in the browser is what you get when you download the raw HTML...but that&#x27;s increasingly less common the case. So now you have to teach how to debug via the console and network panel, on top of basic HTTP concepts (such as query parameters).<p>(Even more problematic is that college kids today seem to have a decaying understanding of what a URL is, given how much web navigation we do through the omnibar or apps, particularly on mobile, but that&#x27;s another issue).<p>I&#x27;ve been archiving a few government sites to preserve them for web scraping exercises [0] (the Texas death penalty site is a classic, for both being relatively simple at first, and being incredibly convoluted depending on what level of detail you want to scrape [1])). But I imagine even government sites will move more toward AJAX&#x2F;app-like sites, if the trend at the federal level means anything.<p>That said, I think the analytics.usa.gov site is a great place to demonstrate the difference between server-generated HTML and client-rendered HTML.<p>But as someone who just likes doing web-scraping, I feel the tools have mostly kept up with the changes to the web. It&#x27;s been relatively easy, for example, to run Selenium through Python to mimic user action [2]. Same with PhantomJS through node, which has vastly improved how accurately it renders pages for screenshots compared to what I remember a few years back<p>[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;wgetsnaps" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;wgetsnaps</a><p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;wgetsnaps&#x2F;tdcj-state-tx-us--death_row" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;wgetsnaps&#x2F;tdcj-state-tx-us--death_row</a><p>[2] <a href="https:&#x2F;&#x2F;gist.github.com&#x2F;dannguyen&#x2F;8a6fa49253c1d6a0eb92" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;dannguyen&#x2F;8a6fa49253c1d6a0eb92</a>

Tbh I didn&#x27;t enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: &quot;I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails &quot;, this just shows a lot of immaturity.

Not wanting to thread hijack, but just going to post an article I wrote a few years back as it covers a few other things that are still relevant; and often still gets referenced. May it&#x27;ll help some people out in combination with OP&#x27;s post.<p><a href="http:&#x2F;&#x2F;jakeaustwick.me&#x2F;python-web-scraping-resource&#x2F;" rel="nofollow">http:&#x2F;&#x2F;jakeaustwick.me&#x2F;python-web-scraping-resource&#x2F;</a>

I wrote a fairly complex spidering and scraping script in Node a few months ago. I found downcache[1] to be absolutely invaluable, particularly as I was debugging my parsing scripts, a I was able to rerun them relatively quickly over the cached responses.<p>However, when the network was no longer a bottleneck, I found that the speed and single-threaded nature of Node became one. It wasn&#x27;t really that slow, relatively speaking, but I had a few hundred gigs of HTML to chew through every time I made a correction, so it was important to keep the turnaround as fast as possible.<p>I eventually managed to manually partition the task so I could launch separate Node scripts to handle different parts of it, but it wasn&#x27;t a perfect split, and there was a fair bit of duplicated work, where a shared cache would have helped a great deal.<p>In retrospect, I should have thrown my JS away and started again in something with easy threading like Java or C#. But -- familiar story -- I&#x27;d underestimated the complexity of the task to begin with, and by the time I understood, I&#x27;d sunk a lot of time into writing my JS parsing code and didn&#x27;t fancy converting it all to another language, particular when it always seemed like &quot;just one more&quot; correction to the parsing would make everything work right. In the end, what was supposed to take a weekend took about three months of work, off and on, to finish.<p>[1] <a href="https:&#x2F;&#x2F;www.npmjs.com&#x2F;package&#x2F;downcache" rel="nofollow">https:&#x2F;&#x2F;www.npmjs.com&#x2F;package&#x2F;downcache</a>

Scraping with Selenium in Docker is pretty great, especially because you can use the Docker API itself to spin up&#x2F;shut down containers at will. So you can spin up a container to hit a specific URL in a second, scrape whatever you&#x27;re looking for, then kill the container. This can be done via a job queue (sidekiq if you&#x27;re using Ruby) to do all sorts of fun stuff.<p>That aside, hitting Insta like this is playing with fire, because you&#x27;re really dealing with Facebook and their legal team.

&gt; AngelList even detects PhamtomJS (have not seen other sites do this).<p>I run a site that aggregates&#x2F;crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I&#x27;ve never felt very good about the fact that I&#x27;m defeating their attempts to block me (even though I feel like I&#x27;m doing them a favor) and will likely retire that bot soon.<p>It kinda sucks that I&#x27;m just grabbing publicly-available content in a very low-bandwidth way, but I really can&#x27;t convince myself that what I&#x27;m doing is very ethical.<p>My to-do list includes making my crawler into a more well-behaved bot and that will have to go.

I don&#x27;t know why more people don&#x27;t use chrome extensions for scraping. Using a boilerplate[1], you can get a scraper up and running in minutes. Start a node server that serves up urls and stores parsed data, and run the scraper in the browser. Best of all, you can watch it running and debug if something goes wrong. I know it doesn&#x27;t scale well if you&#x27;re running a SaaS, but for personal projects and research&#x2F;data normalization it&#x27;s the lowest barrier to entry, in my opinion.<p>[1] <a href="http:&#x2F;&#x2F;extensionizr.com" rel="nofollow">http:&#x2F;&#x2F;extensionizr.com</a>

Sorry guys, hit by traffic - just scaling my EC2 at the moment.

Good stuff.<p>I do a good bit of scraping, and made RubyRetriever[1] to make my life easier but it seems like I&#x27;m getting roadblocked on occasion, probably due to some of the things you mention in your article.<p>Is there any way for a site to verify that only their JS and CSS files are linked? Like preventing injection?<p>[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;joenorton&#x2F;rubyretriever" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;joenorton&#x2F;rubyretriever</a>

At Feedity (<a href="https:&#x2F;&#x2F;feedity.com" rel="nofollow">https:&#x2F;&#x2F;feedity.com</a>), we &quot;index&quot; webpages to generate custom feeds. Over the years, we&#x27;ve designed our system to use a mix of technologies like .NET (C#) and node.js, and implemented a bunch of tweaks and optimizations for seamless &amp; scalable access to public content.

&gt; But if you are automating your exact actions that happen via a browser, can this be blocked?<p>Yes, by checking times between actions and number of actions in a time period, and blocking atypical activity. I was IP banned from a site once for a few months, after trying to scrape it too much and hitting links on the site that were hidden from humans.<p>The random wait settings specified in the post are better than nothing, but still too flimsy. You would need to put hours between requests, only request during a certain 15 hour periods, take days off, and eventually you aren&#x27;t scraping regularly enough to do much good.<p>Scraping is not an API, and I should know- I used to do it for a living. Its unreliable. It requires constant maintenance. APIs can break too, but they are meant for the sort of consumption you are trying for.<p>If you scrape for a living, only do it as a side job.

I actually love Selenium for this purpose, for much the same reasons the author mentions here.<p>It&#x27;s almost impossible for a website to reliably detect that a client web browser is being automated, and I find I can make Selenium scripts much more adaptable to breaking changes in websites when they occur than I can when hooking up my code directly.<p>I actually disagree with the contention that Selenium is slower than directly scraping though. The Firefox driver has always been lightning fast for me and the bottleneck is almost always server requests that would have been necessary either way.

Whilst they mean well, I find this a fundamentally deceptive — the arduous parts of &quot;real world&quot; scraping simply aren&#x27;t in the parsing and extraction of data from the target page, the typical focus of these &quot;scrape the web with X&quot; articles.<p>The difficulties are invariably in &quot;post-processing&quot;; working around incomplete data on the page, handling errors gracefully and retrying in some (but not all) situations, keeping on top of layout&#x2F;URL&#x2F;data changes to the target site, not hitting your target site too often, logging into the target site if necessary and rotating credentials and IP addresses, respecting robots.txt, target site being utterly braindead, keeping users meaningfully informed of scraping progress if they are waiting of it, target site adding and removing data resulting in a null-leaning database schema, sane parallelisation in the presence of prioritisation of important requests, difficulties in monitoring a scraping system due to its implicitly non-deterministic nature, and general problems associated with long-running background processes in web stacks.<p>Et cetera.<p>In other words, extracting the right text on the page is the easiest and trivial part by far, with little practical difference between an admittedly cute jQuery-esque parsing library or even just using a blunt regular expression.<p>It would be quixotic to simply retort that sites should provide &quot;proper&quot; APIs but I would love to see more attempts at solutions that go beyond the superficial.

What bother me the most is that recently I wanted to extract and archive of all the threads I participated in from an Internet forum. The webmaster told me that the BBS he use don&#x27;t provide such a function and that I just had to download each thread manually... (300+ thread in my case).<p>He then say that it don&#x27;t bother him if I scrape theses thread. And I&#x27;m currently figuring out how to manage his site&#x27;s cookie protected search feature, so that my painstaking effort (I&#x27;m not a dev, more a DB guy) could be reproducible more easily by other users of this service.<p>But this shouldn&#x27;t appen in the first place because all post of this service are stored in a cleanly organized MySQl DB. Yet as no method is provided the only way to get back structured data is by scrapping (as the webmaster told me that no, he won&#x27;t run custom SQL because &quot;he don&#x27;t want to mess his DB&quot;).<p>So even if all the data is publicly available through the internet forum only a geek can download a personal archive... or google because google scrape and store everything.

As someone who does a lot of scraping, I was happy to learn about Antigate :)

It&#x27;s trivial to scrape public Instagram URLs...<p><a href="https:&#x2F;&#x2F;github.com&#x2F;kingkool68&#x2F;zadieheimlich&#x2F;blob&#x2F;master&#x2F;functions&#x2F;instagram.php#L421-L428" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kingkool68&#x2F;zadieheimlich&#x2F;blob&#x2F;master&#x2F;func...</a>

Does anybody know what the author means by &quot;lead&quot; (noun)?<p>I don&#x27;t think it&#x27;s any of the regular meanings: <a href="http:&#x2F;&#x2F;www.ldoceonline.com&#x2F;search&#x2F;?q=Lead" rel="nofollow">http:&#x2F;&#x2F;www.ldoceonline.com&#x2F;search&#x2F;?q=Lead</a><p>But it doesn&#x27;t seem to be any of these slang terms either: <a href="http:&#x2F;&#x2F;www.urbandictionary.com&#x2F;define.php?term=lead" rel="nofollow">http:&#x2F;&#x2F;www.urbandictionary.com&#x2F;define.php?term=lead</a>

Have you run into any issues from running all of your scrapers off of AWS, or just from sites detecting that you&#x27;re accessing large numbers of pages in some sort of obvious pattern? I guess I was hoping there would be sites with more interesting ways to screw with web scrapers (rearranging certain page elements or something) than just throwing up a CAPTCHA.

&gt; But if you are automating your exact actions that happen via a browser, can this be blocked?<p>Of course it can! You won&#x27;t be able to defeat even the simplest attempt on anti-scraping based on statistical data. Like even keeping a list of individual rate-limits for &#x2F;16 subnets of actual visiting users and you are in trouble.

Does cheerio account for single page apps? In any case thanks for the tutorial!<p>Anyways I added your stuff here along with other data mining resource:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;kevindeasis&#x2F;awesome-fullstack#web-scraping" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kevindeasis&#x2F;awesome-fullstack#web-scrapin...</a>

To fight scrapers, we show some values as images that look like text (but not all the time)<p>And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors.<p>You can&#x27;t stop them, but you can make their lives painful.

Hooray Melbourne! Would be interested seeing this at a meetup group if you were thinking of presenting.

While everyone is busy debating whether scraping is bad or legal, I just can&#x27;t stop thinking a out Antigate.<p>Of the sweatshops that must have been setup to deliver this service. That, is to me the true horror of this story.

I wonder how effective the CloudFlare anti-scrapper protection is against this approach of breaking captchas.<p>Also, I find it interesting that big websites don&#x27;t just block all traffic from AWS IPs as they do with Tor.

And from the trenches:<p>- rails application<p>- scraping with nokogiri gem on Ruby<p>- simple models doing the scraping in rails app<p>- some scraping is parsed with CSS selectors - nokogiri<p>- some scraping is parsed with regex - nokogiri<p>- persisting to DB, Text, even Google docs<p>- presentation on web, text, pdf, xls<p>Boom

How do you push a button like hit next on a paginated page?

There is so much that&#x27;s missing from this. What about gathering tokens from customers vs. paying for social data feeds? How about canned services like 80legs?

Currently getting 502 Gateway. Guessing this post is also trending on reddit and we hugged it to death :(.

When I was building liisted.com I scraped using Selenium and it worked great.