The talk "How NOT to Measure Latency" [0] taught me all I needed to know in order to start worrying about tail latency in a very well presented way.<p>[0] <a href="https://www.infoq.com/presentations/latency-response-time/" rel="nofollow">https://www.infoq.com/presentations/latency-response-time/</a>
> A common pattern in these systems is that there's some frontend, which could be a service or some Javascript or an app, which calls a number of backend services to do what it needs to do.<p>I think an important idea here is that you should be trying to measure the experience of a user (or as close as possible). If there is a slow service somewhere in your stack, but has no impact on user experience, then who cares? Conversely, if users are complaining that the app feels sluggish, then it doesn't matter if all your graphs say that everything is OK.<p>I find it helpful to split up graphs/monitoring into two categories: 1) if these graphs look fine then the service is probably fine, and 2) if problems are being reported then these graphs might give an insight into <i>why</i> things are going wibbly. In general, we alert on the former and diagnose with the latter. Of course, its nigh on impossible to get perfect metrics that track actual user experience, but we've definitely found it worthwhile to try and get as close as possible to it.<p>---<p>Another fun problem with using summary statistics is they can easily "lie" if the API can do a variable amount of work. For example, if you have a "get updates API" that is called regularly to see updates since the last call, then you end up with two "modes": 1) small amount of time between calls and so super fast and 2) a large amount of time between calls and so is slow. Now, in any given time period the <i>vast</i> majority of the calls are going to be super quick, but <i>every</i> user will hit the slow case the first time they open the app for the first time that day. This results in summary statistics that all but ignore those slow API calls when opening the app.
Another point often missed is the diagnostic value of tail measurements. One of the first things I do at any job is replace the 90th percentile with the maximum in all plots.<p>Sure, it gets messier, and definitely less visually appealing, but the reaction by others has uniformly been "Did we have this data available all along and just never showed it?!"<p>It's also worth mentioning that even in a system where technically tail latencies aren't a big problem, psychologically they are. If you visit a site 20 times and just one of those are slow, you're likely to associate it mentally with "slow site" rather than "fast site".
This is why I don't like require.js, one script requires this script which requires that script. If there is one hiccup somewhere down the line it causes the whole page to have to wait. One of my clients had their website made and wondered why it was so slow (the designers said it needed a faster server) but I found out it was requesting hundreds of .js files in roughly 10 waves. Causing the whole page to take up to 10 seconds to load completely.
A lovely thing about tail latency in the chains that the post talks about is how one service being slow can cascade. Especially in serial chains, when on component is slow, the rest are waiting, in the meantime using resources like memory, sockets, cpu cycles, that could be used to service other requests. In the worst cases, those other services can start responding slowly to other requests, resulting in further degradation.<p>Having circuit breakers and carefully tuning timeouts can help.
> Choosing Summary Statistics<p>This is the #1 thing that people get wrong. It's something that otherwise smart software engineers get wrong because they don't have enough of a background in data analytics.<p>The problem has two parts. One part of the problem is that once you reduce an observation to summary statistics, you can't go back. The other part of a problem is that web services usually generate <i>too much data</i> if you don't summarize.
I have once built an interactive calculator[1] for this exact problem, maybe someone else will find it useful too<p>[1] <a href="https://observablehq.com/@oavdeev/parallel-task-latency-calculator" rel="nofollow">https://observablehq.com/@oavdeev/parallel-task-latency-calc...</a>
"Tail Latency Matters More Than You Think" is more like it.<p>When you see a beach ball or other indicator of delay on your computer you are quite likely to be experiencing tail latency.
This reminded me of the similar cascading effect you get with availability across the aggregate of many services as discussed in "The Calculus of Service Availability": <a href="https://queue.acm.org/detail.cfm?id=3096459" rel="nofollow">https://queue.acm.org/detail.cfm?id=3096459</a>
One complaint here is that serial/parallel is not the way to think about most modern architectures. In modern architectures you are typically working with decoupled event buses which invert the relationship with dependencies. In this case, you become resilient to many negative impacts of tail latency as you're inherently eventually consistent.