How Not to Measure Latency (2015)

<a href="https:&#x2F;&#x2F;archive.ph&#x2F;UiyIl" rel="nofollow">https:&#x2F;&#x2F;archive.ph&#x2F;UiyIl</a><p><a href="http:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20221227140440&#x2F;http:&#x2F;&#x2F;highscalability.com&#x2F;blog&#x2F;2015&#x2F;10&#x2F;5&#x2F;your-load-generator-is-probably-lying-to-you-take-the-red-pi.html" rel="nofollow">http:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20221227140440&#x2F;http:&#x2F;&#x2F;highscalabi...</a>

Needs (2015).<p>I loved the talks from Gil Tene.<p>I always reach for his fork of wrk whenever I need to test throughput:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;giltene&#x2F;wrk2">https:&#x2F;&#x2F;github.com&#x2F;giltene&#x2F;wrk2</a>

I think there&#x27;s times when using a max makes sense and times when using a percentile makes sense.<p>For measurements that are entirely within my own infrastructure, I&#x27;ll always use the max. Outliers there are my responsibility and I want to fix them asap.<p>For measurements that originate from clients, I&#x27;ll always use percentiles. There you&#x27;re measuring the internet, which means people on patchy wifi connections or someone who&#x27;s train just went through a tunnel. The max will always be spiky and there&#x27;s nothing you can do about it.

95% latency seems way more useful as a metric than max() would be. It’s better to spend most of your time improving the experience of most users rather than debugging ultra-rare issues that rarely affect users.

I don&#x27;t remember where I saw it but there&#x27;s also the idea of setting a threshold and asking what percentage of users exceeded it, e.g. how many users are seeing page load times &gt; 1 second.

&gt; Service time is how long it takes to do the work.<p>&gt; Response time is the amount of time spent waiting before the work starts.<p>Is that right? I thought response time was the time from user input to completed response.<p>I&#x27;m not sure (I found parts of the article hard to parse), but I got the impression that author&#x27;s generator won&#x27;t generate a new request until the previous one is complete. That&#x27;s not how I&#x27;d write a generator.<p>Also, it seems odd that the test-rig would discard responses that arrive outside of the test window. Surely it should record all responses to requests that were emitted during the window, even if they arrive outside the window?

Discussed at the time:<p><i>How Not to Measure Latency</i> - <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=10334335" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=10334335</a> - Oct 2015 (23 comments)<p>Maybe we&#x27;ll pinch that non-baity title...

tl;dr:<p>The 95th percentile response time is a bad metric. A typical user executes more than a single request, so it is close to 100% probable that they get a response time above the 95th percentile. Better to tune the maximum response time instead.<p>Edit: When comparing two systems, the Fisher–Tippett–Gnedenko theorem gives guidance about how to do large-sample statistical inference on estimates of maxima: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Fisher%E2%80%93Tippett%E2%80%93Gnedenko_theorem" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Fisher%E2%80%93Tippett%E2%80%9...</a>

I found this article hard to read; it felt like bulleted noted after watching a presentation and many of the statements lacked the context to back them up.<p>The bottom line is that measuring performance is hard, and that you have to be aware that your measurement code and your metrics system have a high likelihood of misleading you if you don’t deeply understand how they work.<p>As a side note, the author didn’t mention my favorite metrics pet peeve, that is the right hand cliff. Often systems will graph a 0 while waiting for a metrics bucket to fill, as when you are aggregating counts over time intervals. This can result in your metric appearing to drop off a cliff in the very recent past.