Today's Outage Post Mortem

As always I'm glad to see Cloudflare post such detailed outage reports. They are one of the few providers I know of that is willing to go into such depth and that is one of the things I appreciate about them. That said, the outage that occurred was one that was indeed fully preventable. We don't exactly have as many locations as they do, but for internal resources at least, not pushing configuration changes to all devices (network included) is pretty standard practice. Basically I imagine for them a good routine to follow might be to script changes so that they are 'rolled out', something along the lines of push manual changes to a scripted 'random' router set (one in country A,B,C), wait 15 minutes and then push to the remaining router sets. That wouldn't work for all situations, such as if the entire network is seeing a DDoS or what have you, but I imagine they could adapt a routine that would prevent this particular scenario.<p>With all of that said, as a Cloudflare customer and also having a call with them tomorrow scheduled already over the WAF stuff, I find it a bit... frustrating that this is occurring now and such a kind of mistake.<p>Edit: As an aside, I wonder if the Puppet module for Junos will be extended to support route statements. That would make this kind of deployment much easier.

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

&#62; CloudFlare currently runs 23 data centers worldwide.<p>Shouldn't that always say - CloudFlare currently runs <i>in</i> 23 data centers worldwide?<p>Or is that just how one would phrase that if you rent multiple racks or a cage in a datacenter? ...because I've seen that a bunch of times before from just about everyone.<p>Just curious.

I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response.<p>It appeals to my limited knowledge and non-existant experience that this would be a solution to the prevention of this occurring again in the future?

To the couldflare folks; It's refreshing to see you take responsibility, but I think you've been a bit too hard on yourselves by taking all the blame. First of all, what you hit was a unknown bug in JunOS, and Juniper is to blame for their part. Using some form of staging to slow roll-out of rule changes <i>might</i> have saved you from a full meltdown, but when you're getting attacked, every second counts. Slow versus fast roll-out is one of those really tough balancing acts in your situation. You did a great job with it; by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again.

That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events.<p>One thing that occurred to me though is that performing a hard reboot of the routers required calling people to physically access the devices and took some time to perform (as you would expect). Although I wouldn't expect it to be needed very often, I'm sort of surprised CloudFlare doesn't have out-of-band remote power cycle capabilities.<p>There may be some factor I'm not considering that would make that an unattractive option, but it does seem like it could cut down an already quick response time even further for any similar events in the future.

if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability.<p>e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites.<p>Hope lessons are learnt and your next generation is less prone to these attacks

Rather unfortunate for the credibility of Cloudfare as a network provider, but you've got to admire them for their honesty and it'll work out better for them in the end. It's amazing how a few lines of code managed to bring down Cloudfare, they could have told us anything and nobody would have been able to question it; instead they gave us the truth and I really respect that. They didn't blame the intern, they didn't blame their hardware or make an excuse about a power outage. In terms of honesty Cloudfare seems to be leading the way regardless of their public credibility or image being tainted. Very impressive response time and resolution of the issue as well, good job Cloudfare!

Wow, that's pretty fast turnaround for a post-mortem (although it looks to have been a simple problem, so easier to figure out what to write)

Two things:<p>1. That video of the BGP routes disappearing is awesome, and<p>2. A 40 minute outage sounds bad, but consider the following timeline (based on the writeup):<p>&#62; T+0: route change made, propagates<p>&#62; T+10: Response team online, attempting local fixes<p>&#62; T+30: Routers across 23 data centres in 14 countries hard reset and networks coming back up.

Funny that now the post mortem is down ...

&#62; Even though some data centers came back online initially, they fell back over again because all the traffic across our entire network hit them and overloaded their resources.<p>I know very little of networking, but this seems to be a recurring pattern that aggravates many major outages. What surprises me is that this so often seems to be a scenario not accounted for.

this is the type of reason why i stopped using cloudflare. there are just too many eggs in one basket. it's as if their entire service becomes a SPOF to your infrastructure.

I think you should have investigated why you got ~90kb packages despite having a max pkg size of ~4kb instead of putting in that rule. :)

&#62; attack packets were between 99,971 and 99,985 bytes long.<p>This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.

Impressive response. 30 minute outage for something most of the hosts I've worked with in the past would have been mystified about for hours. Then a quick RFO and promise of proactive SLA adjustments? Next time I need a CDN or attack mitigation I'll be talking to Cloudflare

What I don't understand is why Cloudflare is making changes to their border routers in the process of protecting their customers. I am a network engineer and I love Juniper, but the reality is with any complex system, every change you make has a possibility of inducing an unexpected failure. I would think Cloudflare would have increased stability by using an architecture where the border routers have a mostly static config, and there is a set of firewalls (e.g. Juniper SRX 5800) behind them that are doing the actual filtering and changing configs in response to threats.

OT: I want to pitch cloudflare for our CDN needs. Can someone estimate the scale of cloudflare wrt. akamai (current provider), in terms of operations, consumers etc.?

<i>Developing good software comes down to consistently carrying out fundamental practices (regardless of the technology)</i> - Paul M. Duvall<p>In this case: Development. Versioned change. Test or staging environment. Tests pass. Production.

So what are they going to change as a consequence? It seems logical to not rely on a single router vendor anymore, or to test new rules on a staging setup at least for a very short time before pushing them to all routers.

Just wondering if the source of the large packets were from a [large] range of hosts or maybe a single host?<p>Ouch if a single host activity took down ~750k websites - whether deliberate and direct or not.

Presumably Jupiter's Junos is closed-source, making investigation more difficult? Do they provide it to some of their bigger clients under an agreement?

I got a "Oops there was a problem" page from Postereous trying to open the blog page/site...

Now we need a Post Mortem on the Post Mortem, as it is now down<p>"Oh noes! Something went wrong."

Posterous seems to be down.

Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash.<p>They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time management of server farms and networks. Of course, in that work, goals included 'lights out data centers', that is, don't need people walking around doing manual work but not the case of 'lights out' as in the CloudFlare outage, and very high reliability.<p>Looking into reliability, that is, putting into a few, broad categories the causes of outages, a category causing a large fraction of the outages was humans doing system management, or as in the words of the HAL 9000, "human error". Yup.<p>And the whole thing went down? Yup: One example we worked with was system management of a 'cluster'. Well, one of the computers in the cluster "went a little funny, a little funny in the head" and was throwing all its incoming work into its 'bit bucket'. So, the CPU busy metric on that computer was not very high, and the load leveling started sending nearly all the work to that one computer and, thus, into its bit bucket and, thus, effectively killed the work of the whole cluster.<p>As one response I decided that real time monitoring of a cluster, or any system that is supposed to be 'well balanced' via some version of 'load leveling', should include looking for 'out of balance' situations.<p>So, let's see: Such monitoring can have false positives (false alarms) and false negetives (missed detections). So, such monitoring is necessarily essentially a case of some statistical hypothesis testing, typically with the 'null hypothesis' that the system is healthy, applied continually in near real-time. So, for monitoring 'balancing', we will likely have to work with multi-dimensional data. Next, our chances of knowing the probability distribution of that data, even in the case of a healthy system, is from slim down to none. So we need a statistical hypothesis test that is both multi-dimensional and distribution-free.<p>So, CloudFlare's problems are not really new!<p>I went ahead and did some work, math, prototype software, etc. and maybe someday it will be useful, but it wouldn't have helped CloudFlare here if only because they needed no help noticing that all their systems around the world were crashing.<p>In our work on AI, at times we visited some high end sites, and in some cases we found some extreme, high up off the tops of the charts, concern and discipline for who, what, or why any humans could take any system management actions. E.g., they had learned the lesson that can't let someone just type in a new rule in a production system. Why? Because it was explained that one outage in a year, and the CIO would lose his bonus. Two outages and he would lose his job. Net, we're talking very high concern. No doubt CloudFlare will install lots of discipline around humans taking system management actions on their production systems.<p>Net, I can't blame CloudFlare. If my business gets big enough to need their services, they will be high on the list of companies I will call first!