Route Leak Impacting Cloudflare

<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20267790" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=20267790</a> is a more recent thread on this.

This appears to be a routing problem. All our systems are running normally but traffic isn&#x27;t getting to us for a portion of our domains.<p><i>1128 UTC update</i> Looks like we&#x27;re dealing with a route leak and we&#x27;re talking directly with the leaker and Level3 at the moment.<p><i>1131 UTC update</i> Just to be clear this isn&#x27;t affecting all our traffic or all our domains or all countries. A portion of traffic isn&#x27;t hitting Cloudflare. Looks to be about an aggregate 10% drop in traffic to us.<p><i>1134 UTC update</i> We are now certain we are dealing with a route leak.<p>@dang etc.: could someone update the title to reflect the status page &quot;Route Leak Impacting Cloudflare&quot;<p><i>1147 UTC update</i> Staring at internal graphs looks like global traffic is now at 97% of expected so impact lessening.<p><i>1204 UTC update</i> This leak is wider spread that just Cloudflare.<p><i>1208 UTC update</i> Amazon Web Services now reporting external networking problem <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a><p><i>1230 UTC update</i> We are working with networks around the world and are observing network routes for Google and AWS being leaked at well.<p><i>1239 UTC update</i> Traffic levels are returning to normal.

There&#x27;s one thing I don&#x27;t understand about this all, it looks like Allegheny Technologies Incorporated (AS396531, a suspected original leaker) was originally announcing 192.92.159.0&#x2F;24.<p>How the heck did their peers not manage to filter a sudden announcement for a range big enough that it managed to snag both 8.8.8.8 and 1.1.1.1. Do upstreams really allow a tiny &#x2F;24 AS to randomly announce a &#x2F;4 and get away with it? Or am I misunderstanding something fundamental about how BGP routes are allowed to propagate?

“AS396531 &quot;Allegheny Technologies Incorporated&quot; is leaking a better-reachable route for AS13335 &quot;Cloudflare, Inc.&quot; towards AS701 &quot;Verizon Business&#x2F;UUnet&quot; explaining the current LSE going on.”<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;OhNoItsFusl&#x2F;status&#x2F;1143117619106652160" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;OhNoItsFusl&#x2F;status&#x2F;1143117619106652160</a>

Final update from me. This was a widespread problem that affected 2,400 networks (representing 20,000 prefixes) including us, Amazon, Linode, Google, Facebook and others.<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;bgpmon&#x2F;status&#x2F;1143149817473847296" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;bgpmon&#x2F;status&#x2F;1143149817473847296</a> <a href="https:&#x2F;&#x2F;twitter.com&#x2F;bgpmon&#x2F;status&#x2F;1143149817473847296" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;bgpmon&#x2F;status&#x2F;1143149817473847296</a>

We&#x27;ve written this incident up: <a href="https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.cloudflare.com&#x2F;how-verizon-and-a-bgp-optimizer-...</a>

As an aside, there is something to be said about the size of Cloudflare when global network problems are reported as being Cloudflare issues.

I’m pretty sure 1.1.1.1 for dns is impacted by this. Initially I thought my WiFi was having issues this morning until I realized it must be dns switching 1.1.1.1 out and besides the cloudflare sites everything is normal again

What&#x27;s weird is that 8.8.8.8 is also intermittently down for me. Are other people having issues with Google DNS too?<p><a href="https:&#x2F;&#x2F;i.imgur.com&#x2F;3ySmVLW.png" rel="nofollow">https:&#x2F;&#x2F;i.imgur.com&#x2F;3ySmVLW.png</a>

Hi Shachar from Peer5 here, we&#x27;re operating a MultiCDN. Cloudflare is actually one of the best performing CDNs. All CDNs encounter issues small to big - that&#x27;s why using multiple providers and intelligently routing between them is critical for high resilience.

The main internet and phone service provider of the Netherlands is down. Even the emergency number (112, our equivalent of 911) is down. Almost everyone is unreachable. The whole telephone network is disrupted.<p>I wonder if it&#x27;s related to this? It does say this kind of BGP thing can be a deliberate malicious attack. Perhaps this? <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;BGP_hijacking" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;BGP_hijacking</a>

@dang etc. Be good if someone changed the title here. 2,400 networks were affected (including parts of Cloudflare, Google, Amazon, Linode, Facebook, ...).

This seems like a partial outage, likely region-based. We have a large number of sites routed through Cloudflare and I can access all of them from home, but our HTTP monitoring software reports the sites as down.

Unless it’s a total coincidence, this looks like this is affecting some Amazon services, including Sagemaker notebooks and Echo devices.

&quot;Cloudflare is observing network performance issues.&quot; not just performance issues, our entire website is unavailable because of it.<p>edit: availability has been alternating between available and unavailable

It’s definitely all countries, just for a specific range of anycast IPs.<p>Our CloudFlare stuff isn’t even pingable. Sometimes it’ll return an echo from a far away DC.<p>It’s been like this for over an hour now and your status page doesn’t even acknowledge it apart from “Network performance issues”.

One of the weirder leaks I&#x27;ve seen, 8.8.8.8 and 1.1.1.1 are both down for me, but everything else is working fine.

What is a route leak?

I realized that it would be an issue like this when downforeveryoneorjustme.com didn&#x27;t load either.

Isn&#x27;t HN on Cloudflare? How are we reading about a CF outage on a site that runs behind CF?

This is affecting far more than just Cloudflare.

I was going back and forth about whether to post this; we&#x27;re clearly experiencing some sort of partial outage, New Relic Synthetics shows everything offline (except for Sydney for whatever reason), but the actual webserver logs indicate healthy traffic and the sites load when I visit it. &quot;Cloudflare is observing&quot; is pretty opaque.

The current it stack needs a do-over. These outages already happen on accident often because of human error. Imagine the damage a state actor could inflict by targeting these large data centers. I hope that some of the newer decentralized cloud startups like dfinity or storj takes over.

It’s caused my site monitoring via PagerDuty to go insane, with texts sent every few minutes.

Seems to be BGP&#x2F;routing related, some networks can access CloudFlare networks normally.

We&#x27;ve been evaluating Cloudflare mainly for doing failovers faster than DNS. This morning I ran some tests to generate graphs to show the typical delay incurred in preparation for a show-and-tell with some key people.<p>I started seeing delays of up to 300 seconds! At best there was a 1 second delay. I wondered if I was going to have present &quot;Why we&#x27;ve decided not to go with Cloudflare!&quot;<p>Any longtime Cloudflare users comment on how rare an event this sort of thing is? It <i>seems</i> rare from eyeballing the recent alert history.

Interesting, isn&#x27;t it, that when it&#x27;s a US based steel plant, it&#x27;s a route leak. When it&#x27;s China Telecom, it&#x27;s a route hijack.<p>The description of it as a leak AFAICT seems to be due to CF getting first dibs on the announcement[†] and positioned it as such. However, I firmly believe that had the general tech press gotten ahead of it first, it still would be treated much more generously than we treat China leaks.<p>[†] grin

How would that kind of disruption happen? Someone else also anycasting the CF IP addresses?

Experiencing around 60% traffic drop on customer&#x27;s sites.

Hey CloudFlare: this page is dependent on ajax.googleapis.com, and if js is disabled, googletagmanager.com. (Also, weirdly, they still have a link to Google+ posts?)

Does anyone know which global sites were or still are unavailable because of Cloudflare crash? Maybe some media sites?

Disabling HTTP proxy and leaving &quot;DNS only&quot; option in CloudFlare DNS settings solved the problem for us.

This morning I&#x27;m finding out just how many of our supporting services rely on Cloudflare as well.

Seeing ~60% drop in traffic here.

Is the specific IP range of the leak known ?

Does anyone know which global sites were unavailable because of Cloudflare crash?