1.1.1.1 outage explanation

This is a great write up. It&#x27;s also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services)<p>A good takeaway from this outage for the average user would be to make sure that your fallback DNS resolvers are operated by totally separate providers. (eg, configure 1.1.1.1 with 8.8.8.8 as a fallback, rather than 1.1.1.1 and 1.0.0.1) (Edit: fixed cloudflare&#x27;s secondary address)

&gt; Our FRP framework allows us to express this in clear and readable code. For example, this is part of the code responsible for performing DNS attack mitigation: &gt; &gt; def action_gk_dns(...): &gt; [...] &gt; &gt; if port != 53: &gt; return None &gt; &gt; if whitelisted_ip.get(ip): &gt; return None &gt; &gt; if ip not in ANYCAST_IPS: &gt; return None &gt; &gt; [...]<p>What does this code sample have to do with FRP? This code seems extremely trivial and doesn&#x27;t give any real indication to me why you&#x27;d need a framework of any sort. It seems like they really want to emphasize that they use FRP, but this code just seems completely unrelated.

Major credit to Cloudflare for publishing a clear, honest, and detailed description of what happened. I wish more companies would do this.<p>One thing I’d be interested to know more about is why it took 17 minutes to fix. While you can and always should strive to make them less likely, outages are inevitable, so how you respond is crucial. Here the outage was very obviously caused by a deployment that I’d assume was supervised by humans – why did it take 17 minutes to roll back?

Great information on the outage. Looks like another version 2 syndrome side effect [1].<p>&gt; <i>Today, in an effort to reclaim some technical debt, we deployed new code that introduced Gatebot to Provision API.</i><p>&gt; <i>What we did not account for, and what Provision API didn’t know about, was that 1.1.1.0&#x2F;24 and 1.0.0.0&#x2F;24 are special IP ranges. Frankly speaking, almost every IP range is &quot;special&quot; for one reason or another, since our IP configuration is rather complex. But our recursive DNS resolver ranges are even more special: they are relatively new, and we&#x27;re using them in a very unique way. Our hardcoded list of Cloudflare addresses contained a manual exception specifically for these ranges.</i><p>&gt; <i>As you might be able to guess by now, we didn&#x27;t implement this manual exception while we were doing the integration work. Remember, the whole idea of the fix was to remove the hardcoded gotchas!</i><p>When porting legacy code it is not only important to understand the edge cases and technical debt built up over time, but to test more heavily in production because you never know if you got them all because some smart guy built them long ago and&#x2F;or there are unknown hacks that were cornerstones of the system for better or worse.<p>Phased and alpha&#x2F;beta rollouts in an almost A&#x2F;B testing way is good for replacement systems. Version 2 systems can also add new attack vectors or other single points of failure that aren&#x27;t as know from legacy problems, the Provision API seems like it is a candidate for that.<p>Over time the Version 2 system will be hardened just before it is EOL and replaced again to fix all the new problems that arise over time. Version 2&#x27;s do innovate but they also shroud fixing old issues and pain points for new unknown problems.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Second-system_effect" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Second-system_effect</a>

What&#x27;s interesting here is that the automatic cure (DDoS protection) was worse than the disease (even if there was an attack, blocking all access to the DNS servers is potentially worse than letting them get overloaded).<p>I wonder if it would be possible to express the idea that if a block being applied drops traffic well below expected levels, it must be a mistake?

Commendable honesty and level of detail in the public RFO.

Understandable outage. I switched back to cloudflare when it came back up, but this did prompt me to drop 8.8.8.8 in as a 3rd fallback.

Doesn&#x27;t seem fixed, still can&#x27;t resolve archive.is cloudflare is giving off cloudflare DNS web errors.<p><a href="https:&#x2F;&#x2F;i.imgur.com&#x2F;APpQPTJ.png" rel="nofollow">https:&#x2F;&#x2F;i.imgur.com&#x2F;APpQPTJ.png</a>

Do they just use python as pseudo code or do they actually run their attack detection in python?

&gt;The next time we mitigate 1.1.1.1 traffic, we will make sure there is a legitimate attack hitting us.<p>I fucking love these guys

This downtime annoyed me at a critical time. Even though I think it&#x27;s great they are transparent about it, I don&#x27;t see myself using their DNS again.

I wonder if the change was manually reverted after 17 minutes, or if Cloudflare has a system that watches for a spike in failures and automatically reverts the most recent change.

Error 1001 DNS resolution error when trying to access archive.li<p>But works with 9.9.9.9

another decent alternative fallback with the same featureset is quad9<p><a href="https:&#x2F;&#x2F;www.quad9.net&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.quad9.net&#x2F;</a>

TL;DR: we should have used an IP that is not traditionally used for testing and internal stuff by everybody including Cisco.

I switched to 1.1.1.1 when it was released and since I’ve had multiple issues with free wifis where they would fail to hijack my dns requests to allow me to login to their portal. I Assume this is a good thing but can someone explain to me why this is happening and what’s the state on improving these wifi portals?<p>PS: sorry for hijacking the thread.

I wonder if they are going to factor the availability into all the blog posts about how fast they are.