Say I have a website that is under DDOS attack. Assume it will be going on indefinitely. Assume my hosting provider has DDOS protection that will protect my servers but at the same time users will be unable to access the website.<p>How can I survive being targeted like this? Cycling IP addresses and updating DNS ad infinitum does not seem like viable solution... because in the end, it's the network attacks that get you, not the application level attacks.
I work for a company in the DDoS mitigation space and there is not nearly enough information in your question for anyone to offer any kind of sensible response for your particular situation.<p>What is your business? How much traffic "normal" do you get? What is the size of the attack? What is the bandwidth of your upstream connection? Who are your customers? Where is it hosted? What are your acceptable thresholds for false negatives and positives? Do you know who is attacking you and why?<p>Most every hosting provider will have some sort of DDoS monitoring and mitigation on their networks already. Their response to sustained or repeated attacks might range from scrubbing the bad traffic before it gets to you and not notifying you at all, to reaching out to you to work with you on both ends of the issue, to cancelling your account.<p>If you just have a fairly simple website that you host yourself, Cloudflare likely a fine option. If you have more advanced needs, you should talk to a more comprehensive DDoS solution vendor.
Assuming this is an API and not a website, my approach would be (other than just paying cloudflare to handle it).<p>Application servers that aren't listed on DNS and their IPs kept secret. A _lot_ of proxy servers that take requests, check the auth, and forward the valid requests to the application servers. For invalid requests, just let them time out.<p>If checking the auth is expensive enough to cause problems, require some proof-of-work in the request too, and only check the auth if the PoW is there.<p>In other words, keep all the DDOS traffic off the application servers. Have sacrificial proxies. Make it cheap to handle the DDOS traffic and expensive to generate the DDOS traffic. And then just process the traffic. Spread the proxies across multiple cloud providers if you need to - but ideally ones with free ingress.
As others have stated, Cloudflare is the best solution here. Other services like AWS WAF will charge you just for validating requests, which means even if the DDOS is mitigated, you're still paying for it to inspect each request and then block it, which now turns it to a denial of wallet attack.
Pay couple of $ for Cloudflare be done with it.<p><a href="https://www.cloudflare.com/ddos/" rel="nofollow noreferrer">https://www.cloudflare.com/ddos/</a><p>Don't forget to have a fresh IP before doing the move so that your origin is unknown.
<i>How to survive indefinite DDoS attack?</i><p>That probably depends on who your customers are. I can chime in one one scenario but maybe this does not help your case. If you are B2B then you could just shut off all public IP's and set up some Anycast VPN WAN accelerated meshes with your customers and use a wildcard sub-domain for each customer to access. They won't like having to set that up on their end but then they will have more resilience, privacy and security than they had before <i>in that they can route around internet problems that would have otherwise caused outages or degraded services</i>. Your company would have to provide professional services to help them with the setup and the customers would need discounts to offset the costs of the hardware assuming they do not already have it. The customers end-user devices would join their network over their VPN, not yours. For some B2B arrangements this isn't even a rounding error and removes scrubbing center costs and removes some SLA breach credits. Give this text to your principal network engineers, approve the spend and they will know what to do.<p>For the customers that do not accept this option, update the contracts and SLA's, then give them a public end-point that has a <i>"best effort"</i> SLA or use a cheap CDN if your contracts permit sending your sensitive data through them. Incentivize them to move to your Anycast VPN WAN accelerated mesh with tighter SLA's and improved security posture by keeping sensitive data off third party data processors like CDN's.<p>[Edit] This would also be a good time to use one of your companies sub-divisions or set up a new legal entity and get a new AS number with some dedicated IPv6 space and let your attackers just keep attacking the old address space. Announce the old IPv4 addresses on a throw away circuit to gather stats for law enforcement and network mailing lists. Try to do as much mesh peering with that new IPv6 space as you can. This is not <i>security through obscurity</i> so to speak, but rather a small piece of The Art of War.<p>The marketing team will not be happy but they can distribute their sites to a myriad of CDN's on marketing specific domains for your company. Gotta keep 'em separated, set some smart cache control headers and hope for the best.<p>An alternative to this semi-complex setup is to pay DDoS scrubbing centers to handle the routing of your datacenter traffic but that can hurt latency sensitive applications. This may be useful for the occasional extortion attack but would be very expensive and sub-optimial(c) in a never-ending indefinite DDoS attack.
There are lots of posts here about using Cloudflare. There's also X4B[1] and Cohmbaton[2] in case you're looking for a non-Cloudflare alternative.<p>You've mentioned that your hosting provider has DDOS protection (which is usually Voxility[3]/Path[4]) will usually send traffic to your VPS through their scrubbing centers, taking care of the L4 attacks. Ratelimiting and aggressive caching can take care of the rest.<p>If you're completely into this on your own, consider setting up a number of PoPs which relay traffic to your backend servers, and these PoPs should drop UDP traffic dropped the prerouting level to drop packets without high CPU consumption. There's also eBPF/XDP, but those require much more engineering work and a compatible network card that can run a eBPF program, otherwise you're still using the CPU to drop packets.<p>[1] <a href="https://www.x4b.net" rel="nofollow noreferrer">https://www.x4b.net</a><p>[2] <a href="https://www.combahton.net" rel="nofollow noreferrer">https://www.combahton.net</a><p>[3] <a href="https://www.voxility.com" rel="nofollow noreferrer">https://www.voxility.com</a><p>[4] <a href="https://path.net" rel="nofollow noreferrer">https://path.net</a>
I'm on Linode since ~2012 and some time ago they added free ddos prottection to their offering, it was advertised on HN too [1]. Hetzner is also offering something to mitigate ddos.<p>There was HN thread asking the same question [2]<p>Having said this, none of my services were targetted by ddos, it made me thinking that maybe maintaining a list of 'healthy' clients IP addresses during good time and then banning anything from outside of that list in bad times? Sounds lame I know, I wait for ideas emerging in this thread :)<p>[1] <a href="https://news.ycombinator.com/item?id=22144369">https://news.ycombinator.com/item?id=22144369</a><p>[2] <a href="https://news.ycombinator.com/item?id=12376596">https://news.ycombinator.com/item?id=12376596</a>
Cloudflare is the simplest solution, but for the sake of treating this like a system design exercise, can you reasonably assume that there is not an infinite supply of devices with which to conduct the attack, fingerprint the devices, and block them?
My question is, what did people do before Cloudflare? There are different types of DDoS attacks. I vaguely remember what we did over a decade ago, which was blackhole IPs. I'm not sure if people still do this directly with iptables, but at the time, this was the bread-and-butter approach for transport DDoS. If your issue was web application related, then you'd move up to fail2ban.<p>DDoS protection for web applications is a different beast than raw transport.<p>If you're a vicim of an amplified DDoS attack, your filtering may need to occur off the server.
I think about this sometimes. Shouldn't there be a way for your app server to report to your DoS protection service which requests are abusive and which are legit? Like a fail2ban but distributed: if someone connects and sends invalid credentials, block them at the ISP or proxy level, not on your host's firewall.<p>It's possible that this doesn't exist for good reasons though, I am not sure how the numbers work out in terms of relative capacity.
People mention Cloudflare. Google Cloud Armor is another option (we looked into both).<p>A colleague was DDoS attacked and was able to move the service behind Cloud Armor in a few hours, effectively ending a ridiculous barrage of network requests. I think CA charges a minuscule amount per request, and can be capped with a fix monthly cost which limits the DDoS liability.
Given the frequent mentions of Cloudflare in this post, I'd like to offer an alternative suggestion.<p>Consider migrating your website to web hosting providers renowned for their DDoS protection, such as OVH or Path. By doing so, you can ensure that your server performance remains unaffected even during ongoing attacks.
Lookup edge service providers like Cloudflare or Akamia. If you’re hosted by a large cloud provider then look into their free and premium edge services offerings. Traffic can be filtered to allow legit users, and consider scaling up servers too if cost of downtime outweighs cost of edge services and scaling up.
What kind of DDOS first, then you need to try to isolate the issue, is it from specific region/country? Can you block an IP range? Sometimes you might have a vulnerable service that allow such attacks, and then look into some providers that might help.
Have you considered placing your website behind Cloudflare? This solution resolved my DDOS issues. Previously, I was regularly dealing dealing with over 20 Gbps and 5M pps.