This has to be one of strangest targets to crawl, since they themselves make database dumps available for download (<a href="https://en.wikipedia.org/wiki/Wikipedia:Database_download" rel="nofollow">https://en.wikipedia.org/wiki/Wikipedia:Database_download</a>) and if that wasn't enough, there are 3rd party dumps as well (<a href="https://library.kiwix.org/#lang=eng&category=wikipedia" rel="nofollow">https://library.kiwix.org/#lang=eng&category=wikipedia</a>) that you could use if the official ones aren't good enough for some reason.<p>Why would you crawl the web interface when the data is so readily available in a even better format?
We're having the same trouble for a few hundred sites that we manage. It is no problem for crawlers that obey robots.txt since we ask for one visit per 10 seconds, which is manageable. The problem seems to be mostly the greedy bots that request as fast as we can reply. So my current plan is to set rate limiting for everyone, bots or not. But doing stats on the logs, it isn't easy to figure out a limit that won't bounce legit human visitors.<p>The bigger problem is that the LLMs are so good that their users no longer feel the need to visit these sites directly. It looks like the business model of most of our clients is becoming obsolete. My paycheck is downstream of that, and I don't see a fix for it.
People got to make bots pay. That's the only way to get rid of this world wide DDOSing backed up by multi billions companies.<p>There are captcha to block bots or at least make them pay money to solve them, some people in Linux community also made tools to combat that, i think something that use a little cpu energy.<p>And in the same time, you offer an api, less expensive than the cost to crawl it, and everyone win.<p>Multi billions companies get their sweet sweet data, Wikipedia gets money to enhance their infrastructure or whatever, users benefits from Wikipedia quality engagement.
Wikipedia provides dumps. Probably cheaper and easier than crawling it. Given the size of Wikipedia it would be well worth a little extra code. it also avoids the risk of getting blocked, and is more reliable.<p>It suggest to me that people running AI crawlers are throwing resources at the problem with little thought.
10 years ago at Facebook we had a systems design interview question called "botnet crawl" where the set up that I'd give would be:<p>I'm an entrepreneur who is going to get rich selling printed copies of Wikipedia. I'll pay you to fetch the content for me to print. You get 1000 compromised machines to use. Crawl Wikipedia and give me the data. Go.<p>Some candidates would (rightfully) point out that the entirety is available as an archive, so for "interviewing purposes" we'd have to ignore that fact.<p>If it went well, you would pivot back and forth: OK, you wrote a distributed crawler. Wikipedia hires you you to block it. What do you do? This cat and mouse game goes on indefinitely.
We need to start cutting off whole ASNs of ISPs that host such crawlers and distribute a spamhaus-style block list to that effect. WP should throttle them to serve like one page per minute.
I was on a panel with the President of Wikimedia LLC at SXSW and this was brought up. There's audio attached <a href="https://schedule.sxsw.com/2025/events/PP153044" rel="nofollow">https://schedule.sxsw.com/2025/events/PP153044</a>.<p>I also like Anna's (Creative Commons) framing of the problem being money + attribution + reciprocity.
The weird thing is their own data does <i>not</i> reflect this at all. The number of articles accessed by users, spiders and bots alike has not moved significantly over the last few years. Why these strange wordings like "65 percent of the resource-consuming traffic"? Is there non-resource consuming traffic? Is this just another fundraising marketing drive? Wikimedia has been know to be less than truthful wrt their funding needs and spent.<p><a href="https://stats.wikimedia.org/#/all-projects/reading/total-page-views/normal|bar|2020-02-01~2025-05-01|(access)~desktop*mobile-app*mobile-web+agent~user*spider*automated|monthly" rel="nofollow">https://stats.wikimedia.org/#/all-projects/reading/total-pag...</a>
I thought all of Wikipedia can be downloaded directly if that's the goal? [0] Why scrape?<p>[0] <a href="https://en.wikipedia.org/wiki/Wikipedia:Database_download" rel="nofollow">https://en.wikipedia.org/wiki/Wikipedia:Database_download</a>
Wouldn't downloading the publicly available Wikipedia database (e.g. via Torrent [1]) be enough for AI training purposes? I get that this doesn't actually stop AI bots, but captchas and other restrictions would undermine the open nature of Wikipedia.<p>[1] <a href="https://en.wikipedia.org/wiki/Wikipedia:Database_download" rel="nofollow">https://en.wikipedia.org/wiki/Wikipedia:Database_download</a>
Working for an open-data project, I am starting to believe that the AI companies are basically criminal enterprises. If I did this kind of thing to them they would call the cops and say I am a criminal for breaking TOS and doing a DDOS, therefore they are likely to be criminal organizations and their CEOs should be in Alcatraz.
> "expansion happened largely without sufficient attribution, which is key to drive new users to participate in the movement."<p>these multi$B corps continue to leech off of everyone's labors, and no one seems able to stop them; at what level can entities take action? the courts? legislation?<p>we've basically handed over the Internet to a cabal of Big Tech
Not sure why these AI companies need to scrape and crawl. Just seems like a waste when companies like OpenAI have already done this.<p>Obviously, OpenAI won't share their dataset. It's part of their competitive stance.<p>I don't have a point or solution. However, it seems wasteful for non-experts to be gathering the same data and reinventing the wheel.
Not just Wikipedia. My home server (hosting a number of not particularly noteworthy things, such as my personal gitea instance) has been absolutely hammered in recent months, to the extent of periodically bringing down the server for hours with thrashing.<p>The worst part is that every single sociopathic company in the world seems to have simultaneously unleashed their own fleet of crawlers.<p>Most of the bots downright ignore robots.txt, and some of the crawlers hit the site simultaneously from several IPs. I've been trying to lure the bots into a nepenthes tarpit, which somewhat helps, but ultimately find myself having to firewall entire IP ranges.
It's not just Wikipedia - the entire rest of the open-access web is suffering with them.<p>I think the most interesting thing here is that it shows that the companies doing these crawls simply don't care who they hurt, as they actively take measures to prevent their victims from stopping them by using multiple IP addresses, snowshoe crawling, evading fingerprinting, and so on.<p>For Wikipedia, there's a solution served up to them on a plate. But they simply can't be bothered to take it.<p>And this in turn shows the overall moral standards of those companies - it's the wild west out there, where the weak go to the wall, and those inflicting the damage know what they're doing, and just don't care. Sociopaths.