> As for fly.io, well, I work there, so, they pay me.<p>Well, that’s nice and all, but if a fly.io customer were attacked with 3.1GB/s throughput, according to the lowest outbound bandwidth price of $0.02/GB [1] they’d be burning at least $3.72/min. 6 times that if attacked from India. That would be a lot less fun.<p>[1] <a href="https://fly.io/docs/about/pricing/" rel="nofollow">https://fly.io/docs/about/pricing/</a><p>Edit: They mentioned they waive charges as a result of attacks: <a href="https://community.fly.io/t/about-rate-limiting/156/4" rel="nofollow">https://community.fly.io/t/about-rate-limiting/156/4</a>
OpenResty (Nginx + LuaJIT) can help you limit the damage of unsophisticated DDoS attacks like these. I keep a count of the requests-per-second I'm getting in each nginx worker. I also set a special cookie for every response from the upstream (it could literally be foo=bar). When the RPS goes approve a certain threshold, if the special cookie is not present, I serve a static HTML page (bypassing the upstream) that sets the cookie and reloads the page (Nginx can do 20K+ RPS without breaking a sweat). In my experience, these fly by DDoS attacks never use cookies, so legitimate users can get through, but the bots are blocked.<p>Of course, if you get hit with something slightly more targeted, this defense is worthless.
Mirror in case the attack successfully picks up again: <a href="https://web.archive.org/web/20220502013024/https://fasterthanli.me/articles/i-won-free-load-testing#" rel="nofollow">https://web.archive.org/web/20220502013024/https://fastertha...</a><p>As a treat, this is a testament to a logic error I made in the caching code (inserted uncachable versions of pages into the cache for a little while). Enjoy!
Regarding the issue of people sending requests directly to your server, bypassing Cloudflare: While you can use IP allowlisting for this, Cloudflare can also be configured to send requests to your server exclusively over mTLS, with a client certificate chaining up to a root CA which they publish. If I were worried about this kind of attack, I'd probably turn that feature on (and then reject requests that don't present a valid client cert); I don't fully trust IP allowlisting. (On the other hand, I can't say for sure that the performance overhead wouldn't be a problem in a DDoS scenario. But it doesn't seem super likely?)
Fun!<p>I tend to be paranoid about exposing things to the Internet, so just put my raw servers behind Envoy. I have tuned that to do rate limiting, circuit breaking (stop sending requests to an upstream when it returns too many errors), idle connection termination, and to shed load when a certain amount of memory is in use, so without any additional configuration for a new service behind the proxy it's somewhat difficult to get the proxy and other services to not respond at all.<p>I'm guessing that in a real attack, the rate limiting service is a weak link. I use a custom rate limit service to aggregate rate limits across a /24 (and hacked that together in an evening), and that is likely the first thing to blow up and erroneously deny service to legitimate users. (I'm sure I have it set up to fail closed, which will be annoying.)<p>I had a hard time ever generating enough load to test any of this for the static serving path. I just set up a mirror of my production environment on my workstation, limited the critical services (Envoy + nginx + rate limit + Redis) to some low amount of CPUs, and then had 31 workers generate synthetic load. I was able to get circuit breakers to open to at least prove that that code works, but I somehow think that I'll run out of network bandwidth before I run out of memory to keep track of open streams. Difficult to load test when the upstream can respond to most requests out of memory.<p>Would be interesting to dig into it more. But for those of you reading this and thinking "I'm going to launch an attack right now", I will just turn off the site if I go over my bandwidth quota. Clone the config repo, host everything locally, run your tests, and send me the results ;)
This inspired me to actually look at my CloudFlare stats, and I realized it's not caching some of my HTML pages even though I had page rules setup to cache everything. I use a static site generator, and it's never been an issue at this point (nobody has any reason to DDoS me). But this is a good motivator to fix that up. I've been procrastinating on fixing a few things on my site, so I'm adding this to the list when I update it. I might also setup the CloudFlare Tunnel at the same time.
It's more fun in my head if "CloudFlare Attack Mode" allows you to wield CloudFlare as a Black-ICE weapon<p><a href="https://en.wikipedia.org/wiki/Intrusion_Countermeasures_Electronics#blackice" rel="nofollow">https://en.wikipedia.org/wiki/Intrusion_Countermeasures_Elec...</a>
I had never considered service back-pressure or circuit breakers until it became a buzz in the java world a few years ago. The concepts are universal though and genuinely interesting building blocks for architecture.<p>Site Reliability Engineering is a fascinating problem space.
Probably doesn't matter much with only a few networks, but this is using the wrong data structure:<p><pre><code> if let Some(net) = ip_nets.load()
.iter()
.find(|net| net.contains(&addr.ip()))
</code></pre>
ip_nets is a 'HashSet<IpNet>' but it should be a radix/patricia tree.<p>Something like <a href="https://lib.rs/crates/iprange" rel="nofollow">https://lib.rs/crates/iprange</a>
Out of curiosity, I googled how much a DDoS attack goes for these days. Apparently they can cost as low as $10/hour. I don't know if the shady people will deliver, but that's what the internet says. So apparently it's pretty easy to DDoS anyone and make it difficult to trace back to you.
> if you want to skip it, search for "After the storm".<p>> Yes, yes, I know, I should add anchor links for headers.<p>This is a rare case where blind people using screen readers have it (a little) easier. Every serious screen reader I know of has a command to skip to the next heading. It's too bad most sighted web users don't have a similar feature handy.
For sites that are mostly read only from non-logged in users Cloudflare can be great. I've got a WordPress content site configured with Cloudflare's APO. Almost 100% of actual human requests for HTML pages or static resources are cached. And there are also a few caching layers at the server level too (setup automatically by Cloudways). The site generates around $15k per month and growing fast on an overprovisioned $50/mo server. I don't think I'll need to spend much more on hosting at even 10x the current traffic due to the caching by Cloudflare. Maybe a bit more for disk space since we store large images but DigitalOcean's Block Storage also makes that incredibly cheap
> That lets me answer questions like "what RSS readers (that aren't browsers) is my audience using?"<p>Very interesting... I also use NNW and FreshRSS
I'm working on two projects that could let a stand-alone Rust web server weather a moderate DDoS attack like the author had:<p>Beatrice [0] - A web server with built-in connection limits and thread limits. It's async but supports non-async request handlers.<p>fair-rate-limiter [1] - In theory, one could use this to shed most of the load from DDoS attacking nodes.<p>[0] <a href="https://crates.io/crates/beatrice" rel="nofollow">https://crates.io/crates/beatrice</a><p>[1] <a href="https://crates.io/crates/fair-rate-limiter" rel="nofollow">https://crates.io/crates/fair-rate-limiter</a>
> The traffic doesn't look like something like headless Chrome was used<p>Setting aside the fact that headless chrome or other browser testbeds do a good job at hiding their presence, what could be the vector for a botnet infection if this were true? Extensions?
> Minutes after I posted this article, the attack resumed. Same shit, different AS.<p>Noob question: was that caused by the article getting posted to HN? Or was it really an attack?
This guy has some great energy.<p>And humor:<p>>> Because it doesn't return an AddrStream but instead a Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>...<p>>> Gesundheit.
I'm interested in what the fly.io would cost if you didn't work there.<p>I got a fly app and can't proxy it through cloudflare cause that doesn't work.
A security audit of my small 6 person office found a Digital Ocean machine trying to brute force one of our Windows machines. I'm becoming less and less impressed with Digital Ocean as time goes on.
I was very surprised to see that your article about golang got flagged here on HN. Never realized that go was such a sensitive topic.
I assume this is related? Can anyone in the golang community give some context?
The code in this article is great advertisement for Golang.<p>Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>? Gesundheit indeed.<p>Great writing as always.
34 million requests really isn't many.<p>Bigger sites might handle that number of requests every second. Hand coded and highly optimized services can handle that number of cached small requests every minute <i>on one machine</i>. After all, that's only an egress rate of a few GBits. And your homepage certainly ought to be both cached and small.