Problems with low DNS TTLs

As a sysadmin with 20+ years experience, I&#x27;ve had long TTLs cause issues on several occasions.<p>I&#x27;ve never regretted a short TTL.

I think it&#x27;d be more interesting to measure the impact on the end user. The article mentions a drop in queries, but aren&#x27;t DNS queries a drop in the bucket compared to the size of most web pages anyway? Is the difference really noticeable?<p>Do you get faster web pages if you cache for a longer time? If you do, shouldn&#x27;t web browsers &quot;soft-invalidate&quot; (use the entry, but update it right after) the cache entry when you&#x27;re just past TTL and &quot;hard-invalidate&quot; (update it before using) after? Do they do that already?

The article claims that web browsers will automatically pick a healthy backend when you return multiple A records, but the behavior doesn&#x27;t seem acceptable to me. I was going to post &quot;I&#x27;ve never seen it work&quot;, but I just tried it and it does indeed work -- the browser hangs for 30 seconds while it waits for the faulty IP address to time out, and then it eventually tries the other IP address, and it does work. (It then retains its selection for a while; I was too lazy to see what happens if I invert the healthiness of the two backends. I also didn&#x27;t try more than 2.)<p>I think most people would call a website down if it was just a white screen for 30 seconds, so while it&#x27;s a nice try on the part of the browsers, you can see why people use short TTLs to get bad backends out of the pool as quickly as possible.

Customer-facing DNS should have TTLs on the order of 15 to 30 minutes. Halving those values to estimate TTL value to the end user, you get 7 to 15 minutes of cached DNS. That&#x27;s about right for most user interactions on the web.<p>Much longer and you run into all the trouble that operators have with keeping DNS accurate. DNS is hard. It is easy to break. And 15 to 30 minutes of waiting is about as much normal human attention span you can apply to a problem that sounds like, &quot;Ok, we&#x27;re all done, is DNS ok?&quot;<p>5 to 10 minute TTLs only benefit operators. Certainly, any TTL less than 5 minutes is an indicator that your operators have no faith whatsoever in their ability to manage DNS.

Once upon a time, I worked in a saas company that would sometimes switch customers to a new instance of a service by switching DNS records -<p>1. Create instance of service running version n+1<p>2. Switch public DNS records to point to new servers<p>3. Wait for TTL to expire<p>4. Turn off old servers<p>(Obviously I&#x27;m simplifying; if nothing else there should be testing steps in there)<p>Unless I&#x27;ve missed something, wouldn&#x27;t the author&#x27;s suggestion to artificially raise the TTL by ignoring the upstream TTL result in the application breaking for customers if they used a DNS resolver that did this?

Maybe for losely coupled systems. Unavoidable in tightly coupled systems because it&#x27;s a convenient way to do things unless you already have elaborate HA infra and protocols in place.<p>For example, if you offer an &quot;entrypoint&quot; that you can guarantee and technically make to be stable, then use longish TTLs. Anycast IPs are an extreme, but inbetween there are many useful modes of exploiting longish but not too long TTLs.<p>On the other hand, if you implement system failover in a locally redundant system and want to exploit DNS so you don&#x27;t have to manage additional technology to make an &quot;entrypoint&quot; HA (VRRP, other IP movements, ...), low TTLs are nice. AWS is I think using 5s TTLs on the ElastiCache node&#x27;s primary DNS names.<p>Finally, 15m max is what I&#x27;m comfortable with. Any longer or much longer, and ANY MISTAKE, and you can easily be in a world of hurt. It&#x27;s no fun sitting out a DNS mistake propagating around the world and the fix lagging behind.<p>And this is only a view on &quot;respectable TTL&quot; values. DNS services like Google&#x27;s public dns probably ignore any or all TTLs for records they pull, and refresh them as fast as possible anyway, at least according to my observation. In that sense, I doubt that most of the internet is still using &quot;respectable&quot; TTLs --- I suspect most systems will RACE to get new data ASAP.

The problem is that the DNS TTL is a feature designed for a static internet of the 70&#x27;s or 80&#x27;s.<p>What this points to is a need for an authenticated DNS pushes for refresh&#x2F;invalidation.<p>All supporting resolvers could keep a list of supporting clients that were told that &quot;foo is at address 42&quot;. If the record changes, the authoritative DNS server sends a DNSSEC signed unsolicited response to all previous requesters to update their records. Obviously the TTL can be extended to keep the cache of requestor IPs reasonably sized.<p>Will this happen? Well, for UDP DNS it depends on DNSSEC, which is already not well supported, and it fixes something that is broken but not terribly so. One could imagine Google arranging this between its DNS resolvers and Chrome, for instance.<p>For DNS over HTTPS, this becomes much more feasible.

&gt;The urban legend that DNS-based load balancing depends on TTLs (it doesn’t - since Netscape Navigator, clients pick a random IP from a RR set, and transparently try another one if they can’t connect)<p>That&#x27;s just not how this works at all. While you could use RR records for this purpose, I believe the author is suggesting that load balancing will happen automatically when the client simply can&#x27;t connect to one of the addresses. That&#x27;s not load balancing. That&#x27;s failover.<p>Additionally, most of the use cases for this that I&#x27;m aware of are Cname -&gt; A record. This is to say, this method is being used with precision rather than RR.<p>I agree that running 60 second TTL&#x27;s regardless of need is inefficient, but at a fast glance, the full argument doesn&#x27;t hold up for me.

This only applies to the first request until the cache expires.<p>If a client makes 50 requests before the cache expires, then those will all be based on the cached result.<p>This is still efficient enough that there&#x27;s probably no more than a single DNS hit for every web page load, even with a short (say, 5 second) TTL, because most web assets will be loaded within that five second window. (If your web page takes longer than 5 seconds to load, you have far more significant issues than a few UDP DNS requests.)<p>Whether the list of invalid use cases are straw man arguments are left as an exercise to the reader, but this article seems to be arguing only one side of the perfectly valid trade-off between flexibility (low TTL&#x27;s) versus latency (high TTL&#x27;s).<p>In other words, if high TTL&#x27;s are so great and there&#x27;s no compelling reasons to not use them, why not make them one year? Ten years?<p>On the other hand, many (probably most) applications can probably absorb a five-minute outage without anyone screaming too loudly.<p>Clearly there is a balance between &quot;long&quot; and &quot;short&quot; (probably somewhere between one second and infinity). It&#x27;s good to think about these things and optimize for lower latency, but if five-minute or longer TTL&#x27;s simply don&#x27;t fit your use case, then don&#x27;t feel bad about it.

I was happy to have a low 10 minute TTL a few days ago when Netlify&#x27;s apex domain IP address stopped working and I had to change it to the new IP that they announced on their status page...! :-) [0]<p>Netlify&#x27;s &quot;previous&quot; IP was down for ~4 hours.<p>[0] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26581027" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26581027</a>

Okay, I thought this would be little more hyperbolic than it is. TTLs under a minute is a little ridiculous. 5m is plenty long for sessions and plenty short for migrations&#x2F;recovery&#x2F;what have you.

LOL lot of arguments for a feature that makes sysadmin&#x2F;dev life easy once a year at the expense of degraded user experience every day (lot of sporadically broken ISP etc DNS servers civilians can&#x27;t be expected to bypass). Digital littering.

I&#x27;ve seen issues with some DNS caches not honouring the TTLs if they&#x27;re too short (less than 1 hour iirc, although memory is a bit hazy, it was some years ago) - in particular academic institutions tended to be the biggest culprits for this.

CloudFlare has a &quot;Auto&quot; TTL option, which is the default, and required to be used when reverse proxying through CloudFlare. There is nothing magical about &quot;Auto&quot; TTL, though: it appears to literally always be 299 seconds. A lot of low TTLs you see are probably caused by CloudFlare.

Doubt long TTLs matter that much, given that plenty of software also has a <i>max</i> TTL value[1], including all popular browsers (Chrome(ium), WebKit aka Safari, Necko aka Firefox, Trident aka IE) and the most popular mobile OS (Android). You could maybe get lucky with some caching on your router, but in my experience cheap consumer routers just act as DNS forwarders and have little to no caching (I could not find any explicit data on this however).<p>1: <a href="https:&#x2F;&#x2F;www.ctrl.blog&#x2F;entry&#x2F;dns-client-ttl.html" rel="nofollow">https:&#x2F;&#x2F;www.ctrl.blog&#x2F;entry&#x2F;dns-client-ttl.html</a>

Part of the problem is that so many devices are poorly behaved when it comes to DNS. At one point I worked for a company that had a large mobile app presence. We setup new authoritative name servers to conduct a test for a week or so. After the test was completed we removed the name servers records. A lot of clients went away very quickly... but way more stuck around way longer than they should have.<p>At two months post test, those test servers were still getting some traffic.

Had to get to the very end to see that &#x27;ridiculously low&#x27; was anything shorter than &quot;between 40 minutes (2400 seconds) and 1 hour.&quot;<p>No thank you, if there&#x27;s an outage that needs a DNS update to resolve it, 5 to 15 minutes is much more reasonable.

If we changed 5 minute TTLs to 1 hour and lost that ability to recover, what would we gain in saved traffic? My guess would be not very much.

Short TTL can be used for activity tracking.<p>You can use dnsmasq --min-cache-ttl= to set the minimum.<p>Unfortunately you have to recompile to have a minimum longer than 1h.

The problem with generalities is that they tend to pick the examples that don&#x27;t generalize well.<p>In the case of github&#x27;s example the author is fixated on DNS where in reality the DNS entry is entry point into fastly&#x27;s anycast CDN endpoints where the DNS is used to point into the general direction of the correct anycast entrypoint. Fastly&#x27;s CTO did a great talk a few years ago about load balancing which addressed the DNS issues based on the actual data they have from the edges that service billions of requests.<p>TL;DR of the DNS portion of that talk is &quot;use as low TTL as you can humanly get away with&quot;

Unrelated to the author&#x27;s post, but for LetsEncrypt TXT records (to have wildcard SSLs), I&#x27;ve always set the TTL very low (in the 1-2 minutes or so range). This is because when I renew SSLs, I don&#x27;t want to wait for DNS caching of those TXT records to resolve all over the Internet.<p>I think that doesn&#x27;t really affect anything traffic-wise. Just a thought I had in mind reading the article.

What is the use cases for having the TTL shorter than 5 minutes?

Reading the article and then reading the comments is interesting. I guess this is a good example of a feature which in theory would benefit both, users and sites - but which falls flat because it&#x27;s infeasible for ops.

Interestingly, in my experience there is always a long tail of laggards after IP changes, where some folks do not notice the change for a very long time or at all. Having a long TTL makes this worse&#x2F;take longer.

I noticed Cloudfront sets a TTL of 60 seconds on its distributions and also on the elastic load balancers. You pay for every Route 53 lookup if you have an ALIAS record pointing there, as is typical. So AWS does not have an incentive to set it any higher.<p>But if I understand it correctly, you can point a CNAME with a long TTL to the appropriate cloudfront.net record, and then you only pay for the CNAME one. The cloudfront.net lookup will not cost you anything. But the latency for your users will be worse because it adds a lookup (because an ALIAS record gets resolved without a lookup).

&gt; The urban legend that DNS-based load balancing depends on TTLs (it doesn’t - since Netscape Navigator, clients pick a random IP from a RR set, and transparently try another one if they can’t connect)<p>Sure but if it can connect but then pukes out on something like a bad ssl or broken app, it’s not going back and trying another host.<p>So, when using dns for load balancing, it’s preferable to have a low ttl with a dns record tied to a host health check. If a host goes unhealthy it takes itself out of rotation, auto scaling brings a new one in, and it’s fully warmed up in a minute.

In 2016 Dyn DNS suffered a DDOS attack and sites including Twitter and Spotify became inaccessible. Higher TTLs would have extended availability from browsers with cached resource records.

One purpose for a low TTL in the solutions I have built is that you want to change the IP. So first you hit the DNS. You get an IP from some main location. Then after the first request you figure out where the user is located. Perhaps spins up some container close to the user. Then on consecutive requests you get an IP much closer to the user.<p>Another usage is to load balance out a lot of users to different web nodes for instance.<p>Edit: spelling

DNS issues could be operated better by many of those running resolvers, for instance, by keeping caches primed for sites to reduce latency to end users - as opposed to extending TTLs.<p>This is probably the cheapest and best solution available for improving DNS related UX issues, and is likely to be something where a commercial DNS provider might do well.

From my short experience, the issue isn&#x27;t that &quot;the new service isn&#x27;t available for the user&quot; but &quot;the new service isn&#x27;t available FOR THE CLIENT&quot;. Cue - &quot;why isn&#x27;t it up yet&quot; emails&#x2F;calls with &quot;it will take up to x hours to propagate&quot;.

Wouldn&#x27;t imposing a lower bound on the TTL push more people to using anycast instead?

So you have a high TTL thinking that DNS servers will cache your IP, yeh right, DNS servers like Google DNS will only cache it for a few minutes. Doesn&#x27;t matter if you have high or low TTL.

I&#x27;ve often wished there was a way a web server could respond to requests with &quot;Your DNS is out of date. Use this IP instead&quot;.

I&#x27;am running local caching dnsmasq with minimum TTL of 1h. Modern internet experience is really awful without it.

I wonder how low TTL compares to browser URL bar queries with respect to impact on DNS user experience.

Author probably never had to switch servers because of failure etc and then had to wait 24 hours until the traffic came back up while losing money and getting angry emails from clients who e.g. bought advertising.

It doesn’t sound like the author has ever operated a large scale service. There are reasons why every big operator has short TTLs and it isn’t because they are stupid.