This still isn't what I'd call "safe". Retries are amazing at supporting clients in handling temporary issues, but horrible for helping them deal with consistently overloaded servers. While jitter & exponential backoff help with the timing, they don't reduce the overall load sent to the service.<p>The next step is usually local circuit breakers. The two easiest to implement are terminating the request if the error rate to the service over the last <window> is greater than x%, and terminating the request (or disabling retries) if the % of requests that are retries over the last <window> is greater than x%.<p>i.e. don't bother sending a request if 70% of requests have errored in the last minute, and don't bother retrying if 50% of the requests we've sent in the last minute have already been retries.<p>Google SRE book describes lots of other basic techniques to make retries safe.
This is one of those things that sort of exposes our industry maturity versus other engineering that's been around longer. You would think by now that the various frameworks for remote calls would have standardized down to include the best practice retry patterns, with standard names, setting ranges, etc. But we mostly still roll our own for most languages/frameworks. And that's full of footguns around DNS caching, when/how to retry on certain failures (unauthorized, for example), and so on.<p>(Yes, there should also be the non-abstracted direct path for cases where you do want to roll your own).
Summary of the article: use exponential backoff + jitter for retry intervals.<p>What author didn’t mention: sometimes you want to add jitter to delay the first request too, if the request happens immediately after some event from server (like server waking up). If you don’t do this, you may crash the server, and if your exponential backoff counter is not global you can even put server into cyclic restart.
Really nice animations, I especially liked the demonstration of the effect that after some servers will "explode", any server that will be restarted will automatically be DoS'ed until we'll throw a bunch of extra temporary servers into the system. Thanks.
The animations are so cool!!!<p>In general the phenomena is known as _metastable failure_ that could be triggered when there are more things to do during failure than normal run.<p>With retry, the client do more work within the same amount of time, compared to doing nothing or doing exponential backoff.
For a lot of things, retry once and only once (at the outermost layer to avoid multiplicative amplification) is more correct. At a large enough scale, failing twice is often significantly (like 90%+) correlated with the likelihood of failing a third time regardless of backoff / jitter. This means that the second retry only serves to add more load to an already failing service.
This is the client side of things. And I think this is a great resource that everyone who writes clients for anything, should see.<p>But there is an additional piece of info everyone who writes clients needs to see: And that's what people like me, who implement backend services, may do if clients ignore such wisdom.<p>Because: I'm not gonna let bad clients break my service.<p>What that means in practice: Clients are given a choice: They can behave, or they can<p><pre><code> HTTP 429 Too Many Requests</code></pre>
I worked at a company with a self-inflicted wound related to retries.<p>At some point in the distant (internet time) past, a sales engineer, or the equivalent, had written a sample script to demonstrate basic uses of the API. As many of you quickly guessed, customers went on a copy/paste rampage and put this sample script into production.<p>The script went into a tight loop on failure, naively using a simple library that did not include any back-off or retry in the request. I'm not deeply familiar with how the company dealt with this situation. I am aware there was a complex load balancing system across distributed infrastructure, but also, just a lot of horsepower.<p>Lesson for anyone offering an API product: don't hand out example code with a self-own, because it will become someone's production code.
I have been thinking about queueing theory lately. I don't have the math abilities to do anything deep with it, but it seems like even basic applications of certain things could prove valuable in real world situations where people are just kind of winging it with resource allocation.
If a picture is worth 1,000 words, then what's a well made animation worth? These are great intuitive representations of your retry methods. Bravo!