Cloudflare outage on July 17, 2020

Head of DevOps at a major financial exchange where latency &amp; resiliency is at the heart of our business, and yes, we pay Cloudflare millions. I see two things here:<p># Just be ready<p>Most definitely not the first time Cloudflare has had trouble, just like any other system: it will fail eventually. If you&#x27;re complaining about the outage, ask yourself the question: why were not you prepared for this eventuality?<p>Spread your name servers, and use short-TTL weighted CNAMEs, defaulting to say, 99% Cloudflare, 1% your internal load balancer. The minute Cloudflare seems problematic, make it 0% 100% to bypass Cloudflare’s infrastructure completely. This should be tested periodically to ensure that your backends are able to scale &amp; take the load without shedding due to the lack of CDN.<p># Management practices<p>Cloudflare&#x27;s core business is networking. It actually embarrasses me to see that Cloudflare YOLO&#x27;d a BGP change in a Juniper terminal without peer reviews and&#x2F;or without a proper administration dashboard, exposing safe(guarded) operations, a simulation engine and co.? In particular, re-routing traffic &#x2F; bypassing POPs must be a frequent task at scale, how can that not be automated so to avoid human mistakes?<p>If you look at the power rails of serious data centers out there, you will quickly notice that those systems, although built 3x for the purpose of still being redundant during maintenance periods, are heavily safeguarded and automated. While technicians often have to replace power elements, the maintenance access is highly restricted with unsafe functions tiered behind physical restrictions. An example of a common function that&#x27;s safeguarded is the automatic denial of an input command that would shift electrical load onto lines beyond their designed capacity - which could happen by mistake if the technician made a bad assumption (e.g. load sharing line is up while it&#x27;s down) or if the assumption became violated since last check (e.g. load sharing line was up when checked, became down at a later time - milliseconds before the input even).

CloudFlare is a good company and everyone has outages. IMHO the post-mortems they post are not only some of the best I&#x27;ve read from a big company, but they are produced quickly.<p>I only wish they could update cloudflarestatus.com more quickly. Shouldn&#x27;t there be some mechanism to update that immediately when there is an incident? When the entire internet knows your down and your status page says All Systems GO! it looks very poorly on them.

Here&#x27;s our blog post detailing what happened today and the mitigations we&#x27;ve put in place to ensure it doesn&#x27;t happen again in the future: <a href="https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;cloudflare-outage-on-july-17-2020&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;cloudflare-outage-on-july-17-202...</a>

I was on a call with an investor and an employee mouthed to me, silently, &quot;everything is down!&quot;<p>Immediate hot flash.<p>After I got off the call (thank god he had an appointment to run to), I checked it out. Our internal dashboards were all green so we realized it was a DNS issue pretty quickly.<p>Since we couldn&#x27;t get into Cloudflare we searched Twitter and realized it was their issue and I stopped worrying.<p>One of the benefits of CF and other major internet vendors is that when they&#x27;re down, you can kind of shrug it off and tell your customers to wait a bit. Not so if you&#x27;re using a smaller&#x2F;unknown CDN company.

Compare this to Facebooks SDK “postmortem” and you can tell which company cares more about its customers.

It&#x27;s seems these major infrastructure outages always are from a configuration change. I remember Google had a catastrophic outage a few years ago, and the postmortem said it all began as a benign configuration update that snowballed worldwide. In fact I tried googling for it and found the postmortem of a more recent outage, also from a configuration change.<p>Some seasoned sysadmin will say to me, &quot;Of course it&#x27;s always from a configuration change. What else could it be?&quot; I don&#x27;t know, it seems like there are other possible causes. But in today&#x27;s superautomated infrastructures, maybe config files are the last soft spot.

&gt; We saw traffic drop by about 50% across our network. Because of the architecture of our backbone this outage didn’t affect the entire Cloudflare network and was localized to certain geographies.<p>I&#x27;m not even sure.. Is that second sentence supposed to signal some sort of success? Dropping 50% of your traffic isn&#x27;t isolated. If your gonna try to spin it, at least bury the damn lede. Further:<p>&gt; The affected locations were San Jose, Dallas, Seattle, Los Angeles, Chicago, Washington, DC, Richmond, Newark, Atlanta, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, St. Petersburg, São Paulo, Curitiba, and Porto Alegre. Other locations continued to operate normally.<p>Locations with THEIR equipment, but certainly not all &quot;affected&quot; locations. I live 4 hours from Dallas and can assure you that I was impacted. That coverage is like.. Most of the United States, Europe, Brazil and who knows how much of South America? Oh right, 50% of their traffic!

Wow, BGP brings down globally-used DNS. It’s like a perfect lesson in weak points of the modern web’s accidental design.

Network Engineering: Invisible when you’re killing it at your job, instantly the enemy when you make a mistake.

We need to do something about BGP.<p>Just in the past year Verizon, IBM, Apple, now Cloudflare have seen outages from BGP misconfiguration. The Verizon issue took down a significant part of the internet.<p>BGP is a liability to society. We need something which doesn&#x27;t constantly cause widespread outages.

In the last three years we have hosted our enterprise software on Azure, and the only outages we&#x27;ve had have been caused by mistakes or issues at Cloudflare. Azure has been rock solid but our customers don&#x27;t understand that and assume that we&#x27;re &quot;just down&quot;, which impacts our SLAs.<p>During the most recent outage a few weeks ago, Azure were available to discuss the issue by phone. I wish I could say the same for Cloudflare.<p>I would be interested to hear from anybody who knows of a good alternative to Cloudflare. I&#x27;m completely fed up with them.

&gt;&quot;We are making the following changes:<p>Introduce a maximum-prefix limit on our backbone BGP sessions - this would have shut down the backbone in Atlanta, but our network is built to function properly without a backbone. This change will be deployed on Monday, July 20.<p>Change the BGP local-preference for local server routes. This change will prevent a single location from attracting other locations’ traffic in a similar manner. This change has been deployed following the incident.&quot;<p>It should be noted that configuring prefix limits for your BGP peers is kind of BGP 101. It&#x27;s mentioned in every &quot;BGP Best Practices&quot; type document.[1] It&#x27;s there for exactly this purpose to prevent router meltdown and resource exhaustion. For a company who blows their horn as much as these folks seem to about their network this is embarrassing.<p>I think it&#x27;s worth mentioning that it was this time last year when Verizon bungled their own BGP configuration and brought down parts of the internet. When that incident occurred Cloudflare&#x27;s CEO was front and center excoriating them for accepting routes without basic filtering [2]. This is exact same class of misconfiguration that befell them yesterday.<p>[1] <a href="https:&#x2F;&#x2F;team-cymru.com&#x2F;community-services&#x2F;templates&#x2F;secure-bgp-template&#x2F;" rel="nofollow">https:&#x2F;&#x2F;team-cymru.com&#x2F;community-services&#x2F;templates&#x2F;secure-b...</a><p>[2] <a href="https:&#x2F;&#x2F;twitter.com&#x2F;eastdakota&#x2F;status&#x2F;1143182575680143361?lang=en" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;eastdakota&#x2F;status&#x2F;1143182575680143361?la...</a>

Commented on the original outage: Hopefully with this outage Cloudflare will provide non-Enterprise plans a CNAME record, allowing us to not use Cloudflare DNS and more quickly bypass Cloudflare if the need arises.

2 years ago, about 1 in 3 people in the UK were watching England in the World Cup.<p>Towards the end of the game, a CDN the bbc used crashed, taking a million people’s live streams offline.<p>Traditional TV with its 20 million plus viewers worked fine.<p>A 15 minute global outage during the World Cup or Super Bowl is not acceptable in the world of boring old TV<p>Meanwhile github has been down how many times this year?<p>IT is still a terrible industry of cowboys. It’s just hidden under the veneer of abstaction, microservices and outsourcing. Other industries like the national grid or water of radio have outages that affect a local geographic area or a limited number of people, but they are far more distributed than the modern internet. It’s ironic a network designed to survive nuclear war can’t survive a typo.<p><a href="https:&#x2F;&#x2F;m.huffingtonpost.co.uk&#x2F;entry&#x2F;bbc-iplayer-crashes-in-final-minutes-of-england-v-sweden-in-the-world-cup_uk_5b40e196e4b09e4a8b2d88fc" rel="nofollow">https:&#x2F;&#x2F;m.huffingtonpost.co.uk&#x2F;entry&#x2F;bbc-iplayer-crashes-in-...</a>

what is funny you go to cloudflare customers page, check all these companies status page, all down, non of them admits it&#x27;s due to third party cloud provider e.g. cloudflare. In most cases it was &quot;performance issue&quot;. Its so silly... you&#x27;re in the interconnected world. It&#x27;s ok your major cloud providers went down...

Why did it take so long for a status page to be published I wonder?<p>From the timeline in the blog post the issue with Atlanta was fixed between 21:39 to 21:47 but a status page wasn&#x27;t published until 21:37. Everything had been broken for over 20 minutes at that stage with lots of people already posting about it or other status pages reflecting issues. See <a href="https:&#x2F;&#x2F;twitter.com&#x2F;OhNoItsFusl&#x2F;status&#x2F;1284239769548005376" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;OhNoItsFusl&#x2F;status&#x2F;1284239769548005376</a> or <a href="https:&#x2F;&#x2F;twitter.com&#x2F;npmstatus&#x2F;status&#x2F;1284235702540984321" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;npmstatus&#x2F;status&#x2F;1284235702540984321</a><p>Without an accurate status page it leaves businesses pointing the finger everywhere wondering whether its their hosting provider having issues, their CDN, DNS provider etc etc.

You mean the service that everyone is centralizing in caused problems because everyone centralized in it? Pikachu shock face. If you&#x27;re web or network dev act responsibly. Don&#x27;t just pick cloudflare because everyone else does. Don&#x27;t pick cloudflare <i>because</i> everyone else does.

I got the same issue today when I was on-call. took me 1 hour to figure out it was Cloudflare.<p>I&#x27;m currently working on a project to monitor all the 3rd party stack you use for your applications. Hit me up if you want, access I&#x27;ll give free access for a year+ to some folks to get feedbacks.

Cloudflare is experiencing a bit of karma and a bit of Murphy’s Law.<p>They slung some mud not long ago and now it came to bite them. They were a bit righteous on their reliability. However, anyone in this game long enough knows it’s only a matter of time before shit goes down. If they didn’t have any graybeards over there to tell them this, then hopefully they earned some gray.<p>Stuff was down long enough for Googles and OpenDNS caches to expire, and to take down DigitalOcean in some respects.<p>Thankfully CF can afford to learn and make improvements for the future. Not all organizations are that lucky.

This is not the first time human error on BGP routing configuration which then caused a significant portion of the Internet down. Is there any kind of configuration validator that can be implemented to prevent and catch this type of errors? I am fairly sure this won&#x27;t be the last time we will hear about human error on BGP routing config causing Internet down.<p>Or is BGP intrinsically a unsafe protocol without builtin protections on this sort of human mistakes?

It would be interesting to classify network outages and determine the number that involved practices that would be obviated by a standard VCS &#x2F; release process like found in software. Routers&#x2F;firewalls seem to be a particular pain point everywhere.

I guess it&#x27;s easy to become complacent when you&#x27;re a networking expert at Cloudflare and likely making several of these ad-hoc, on-the-fly config changes every single day, but it&#x27;s always good to remember why Juniper introduced the automatic rollback feature.<p>Of course, this particular outage would not have been <i>prevented</i> even if they had used<p><pre><code> # commit confirmed </code></pre> as it can&#x27;t stop you from screwing up but it almost certainly would have limited the duration of the outage to ~10 minutes (plus a minute or two, perhaps, for the network to recover after the rollback) -- and it could have been shorter than that had they used &quot;commit confirmed 3&quot;, for example.<p>Even as a lowly network engineer working on networks much, much smaller than Cloudflare&#x27;s, for pretty much my entire professional career, it&#x27;s been my standard practice to start off pretty much <i>ANY</i> change -- no matter how trivial -- with<p><pre><code> # commit confirmed 3 </code></pre> or<p><pre><code> # reload in 3 </code></pre> or similar, depending on what type of gear I was working on (and, of course, assuming the vendor supported such a feature).<p>This applies even when making a changes that are so &quot;simple&quot; that they just &quot;can&#x27;t&quot; go wrong or have any unexpected or unintended effects.<p>In fact, it applies <i>ESPECIALLY</i> in those case! It&#x27;s when you let your guard down that you&#x27;ll get hit.<p>---<p>Fortunately, all that was necessary (I assume) to recover in this case was to<p><pre><code> # rollback </code></pre> to the previous configuration. Then, the correct configuration could be made. That still had to be done manually, however, and resulted in a 27 minute outage instead of what could have been a 5 or 10 minute outage.<p>I would hope that Cloudflare has full out-of-band access to all of their gear and are able to easily recover from mistakes like this. If they had lost access to the Atlanta router and weren&#x27;t able to log in and revert the configuration manually, this outage could have lasted much, much longer.

Did this sink any eCommerce websites?

Everyone has outages, CloudFlare is a decent company making a good product.<p>BUT<p>What&#x27;s interesting here is that so many non-CloudFlare services went down (including even AWS - partially) caused by DNS outage - because every sysadmin and his mom are using 1.1.1.1 as their DNS.

looks like Tesuto was a good buy<p><a href="https:&#x2F;&#x2F;investors.fastly.com&#x2F;news&#x2F;news-details&#x2F;2020&#x2F;Fastly-Achieves-100-Tbps-of-Edge-Capacity-Milestone&#x2F;default.aspx" rel="nofollow">https:&#x2F;&#x2F;investors.fastly.com&#x2F;news&#x2F;news-details&#x2F;2020&#x2F;Fastly-A...</a>:<p>&gt; By emulating networks at scale, Tesuto’s technology can be used to create sandbox environments that simulate the entire Fastly network, providing a view into the potential impact of a deployment cycle before it is put into production.

It&#x27;s fine that they made mistakes and there were an outage, shit happens to everybody.<p>What is really scary though that half of the internet stopped working. That&#x27;s not ok!

It&#x27;s amazing to me that in this situation, humans still had to intervene and update a configuration file.<p>I&#x27;m surprised stuff like this wouldn&#x27;t happen more often and there would at least be a well tested, automated remediation step in place which also validates the change prior to going live.<p>I get they may be busy solving other issues, but it&#x27;s interesting this isn&#x27;t a more fool proof procedure given the huge impact a mistake can have.

I wish they would go into why the rather complete outage was not visible to cloudflarestatus.com. I fully understand that mistakes can be made, but I&#x27;m really not pleased with how hard it was to tell if I was experiencing a localized issue. During the entire outage, cloudflarestatus.com displayed &quot;all systems operational&quot; for me, once accessed it with a functioning DNS resolver.

Fun, I got hit by this since I use cloudflared behind my Pi-Hole. I was able to troubleshoot the issue, localized the cause to Cloudflare, found the partial outage in various regions and assumed I was affected, switched to using Level 3 DNS temporarily. I&#x27;m glad to see it&#x27;s back up and this is a great retrosepctive.

I don&#x27;t understand Cloudeflare is making direct configuration changes to a router like this. If these are changes that are made regularly, why not use a tool to make them. You can then assure that only certain changes are possible, preventing simple mistakes like this.

Hmm, duckduckgo and qwant also seem to have some trouble, time to head for <a href="https:&#x2F;&#x2F;searx.space&#x2F;" rel="nofollow">https:&#x2F;&#x2F;searx.space&#x2F;</a> I guess?

Fascinating to see that airport codes are used as datacenter names. I know at least one other company which does that but I thought that&#x27;s something peculiar to them.

Is there any work being done to replace BGP or current IGP&#x27;s? Wondering if modern computing and memory capacity and algorithms can be used to make more fail-safe protocols.

Random question:<p>The Cloudflare revolvers definitely went down (1.1.1.1 and 1.0.0.1), do we know if authoratative DNS did?

Large portion of some network is down? Oh, right, it&#x27;s BGP. It&#x27;s always BGP.

What configuration error? Was this human or automatic? What was done to mitigate?

It shows that the system is pretty fragile.