Basecamp was under network attack

Some great language there: framing it as an attack by criminals (gains sympathy from users), explains in plain-terms what a DDOS is (front door analogy), emphasizes (twice!) that user data is safe, apologizes for the likely downtime, informs people where to get updates.<p>Probably worth bookmarking this for when you [hopefully never] have to deal with this same situation.

I take it at one point people will start to believe that I work for OVH (I really don&#x27;t) but... OVH has a mandatory DDoS protection on all its dedicated servers: fees have been slightly raised to take that mandatory protection into account.<p>There are a few gotchas, including if I understand it correctly the need to &quot;retry twice&quot; when you try to SSH in your server when a DDoS is going on but...<p>OVH doesn&#x27;t even feel a 85 Gbps attack (let alone a 20 Gbps one like in the article). They can deal with attack much larger than that automatically.<p>They seem to have very good DDoS protection against the &quot;flood&quot; type of DDoS. And this is pretty much transparent to users.<p>I hope more and more hosting company start implementing similar anti-DDoS features: more competition would bring better protection against flood-type DDoS and cheaper price.<p>Here&#x27;s the explanation as to how their system works (in french but there are several graphics):<p><a href="http://www.ovh.com/fr/a1164.protection-anti-ddos-service-standard" rel="nofollow">http:&#x2F;&#x2F;www.ovh.com&#x2F;fr&#x2F;a1164.protection-anti-ddos-service-sta...</a><p>Basically as soon as a DDoS trying to saturate your server(s) is detected the attacker faces the problem of needing to DDoS... OVH itself.<p>And the DDoS doesn&#x27;t even make it to your server while the legitimate trafic still does.<p>I find it great that there are people actually looking for solutions to the DDoS issue.

We got hit by a DDoS about a year ago. Rackspace (who normally has amazing support) quietly null routed us and went about their day. No heads-up, trouble ticket, or any other form of notification. They didn&#x27;t even put a note in our account so when we contacted their support to figure out why our servers were unresponsive outside their network the poor guy who answered the phone was just as confused as I was.<p>We&#x27;ve taken some steps since then to hopefully reduce our vulnerability. I&#x27;d be really interested in a DDoS protection best practices guide for small SaaS businesses.

I&#x27;ve had really negative experience with these type of criminals.<p>I was hired as a CEO at an &lt;unnamed&gt; company ($200m+ revenue) and we were hit by this type of attack.<p>Every second of being down cost us literally $10k, so we quickly negotiated with criminals for $5k one time payment and they stopped the attack.<p>Unfortunataly a few weeks later we were hit by 3 new attacks. Apparently the word had spread and these new attackers demanding $50k.<p>We were not going to pay $50k but I was also unable to stop the attacks. I was let go a few days later as we had a down time of 2 days and I wasn&#x27;t able to fix this problem.<p>Crap.

Although a smaller service, we were in a similar situation a couple of years ago. We assumed it was a competitor because there were not monetary requests, just a massive DDoS via torrents that lasted almost a week. Data center didn&#x27;t help us in any way... it was crazy. Worst thing is that 90% of customers have no clue what a DDoS is and how hard it is to handle.

Is it just me or are these attacks becomming more and more common? I hope we can get some more details on the attack like the origination of it, type used, and what steps were take to mitigate it. I always use information like this as a learning opportunity :)

Has anyone defended a DDoS attack on an application hosted on Amazon&#x27;s AWS&#x2F;EC2?<p>If so, how did that go?<p>Did Amazon help?

What law enforcement do you call in these situations. I imagine it would be a waste to call local police.<p>I don&#x27;t know how you would get feds to pay attention?

Would CloudFlare help here?

Does anybody know how many companies, upon receiving a blackmail &quot;give us $300 or you&#x27;ll be DDoSed&quot; email, pay it? For every meetup.com or Basecamp that resist, how many actually give in to the blackmailer&#x27;s demands?

I&#x27;m wondering what happens to botneted subscribers from which the attacks originate. Is any attempt made to locate them and contact their ISPs? I think there should be, and subscribers found to be participating in the attack (presumably unknowingly) should be disconnected immediately. After all it&#x27;s the subscribers&#x27; responsibility to keep their computers botnet free. Launching a DOS attack, even unknowingly, is probably violating the contract they signed with their ISP.

Crime, crime, crime, criminal. While technically (and probably also morally) true, was I the only one to find the emphasize weird?

Whoever is doing this thank you for reminding me how important Basecamp is to my business. I hope they hunt you down.

A speculative thought:<p>Apart from being distributed, the insidious power of DDoS appears to lie in &quot;subscriber-calling-server&quot;. Why not go the other way around? At least only for specific subscription services, not general purpose web access.<p>The situation of a DDoS attack is first communicated by the web service provider texting a subscriber, who texts back their present IP address. The web service provider then &quot;calls&quot; the subscriber from a hitherto unknown IP address. Of course, that address could be leaked too, but at least it&#x27;s not obvious public knowledge like a DNS entry.<p>Sounds like circuit switched telephony&#x2F;modems rather than packet switching, but can it be implemented in software?

How do larger companies (like Basecamp) prepare for these kinds of risks? Do they contract with DDoS mitigation firms beforehand, or do most tend to hire help only when they are actually attacked?

Something along the lines of CloudFlare could be an option here. However, if the attacker does indeed know the actual IP of the Bootcamp servers (and Bootcamp allows traffic from IPs other than CF) that point is moot.<p>Set up CF, only allow traffic from CF.<p>On another note, having CF monitor an attack like this could help them do more research into mitigating these attacks in general and allow them to try and hunt the attacker. They tend to make things like this public which would benefit everyone.

I wonder if there will be a day where on-premise solutions will be touted as the solution to the DDoS vulnerability of cloud-based solutions, in much the same way that there seems to be an ebb and flow between fat and thin clients over the course of computing history.

Is there something like cloudfare but more aggressive?<p>Like something that tries to find exploits on the machines used in the attack and try to shut them down, close their internet connection or inject a self-targeting DNS or something of the sort?

&gt; When these attacks happen, the rest of the internet will sometimes put you in quarentine to prevent the fire from spreading.<p>I&#x27;m interested about what he means by quarantine.<p>Does it mean that ISP&#x27;s will stop accepting traffic going to their servers?

Every business experiences fires that they have to put out, and their transparency on what exactly the issue is keeps us informed and on their side.

We need the kind of concerted attention paid to this stuff that we gave to horse thieves in the Old West.

This is another great example of why I wish there was support for disabling commenting on gists.

Forget baecamp. Setup a webserver throw Colalbtive on it. Now you are in control of your data (you are now also responsible for the uptime).<p>Colabtive: <a href="http://collabtive.o-dyn.de/" rel="nofollow">http:&#x2F;&#x2F;collabtive.o-dyn.de&#x2F;</a>

they did get a blackmail email so it does seem like they are being targeted by someone.

is it the first time they are facing this sorta attack ?

Yet another reason we should be utilizing P2P WAY more often

A perfect time for those affected to test drive BaseCamp&#x27;s competitor <a href="https://www.teamwork.com/" rel="nofollow">https:&#x2F;&#x2F;www.teamwork.com&#x2F;</a>