Surprising Economics of Load-Balanced Systems

This can be flipped into a simple heuristic known as the &quot;square root staffing rule&quot; (sometimes &quot;... law&quot;). The name comes from call centres.<p>Basically, the number of servers you need to serve X amount of demand with Y probability of queuing is not linear with X. It is proportional to the square root of X.<p>The intuition is that a call centre agent is either talking to a customer, or they are not. If they are not talking to a customer, then they can immediately serve a customer. The odds that a given agent is talking to a customer is on a probability distribution, usually assumed to be the normal distribution.<p>In a normal distribution, larger samples lead to more and more appearances of outliers in the sample. More agents means more &quot;outliers&quot;, in this case, more idle agents. And that curve is not <i>linear</i>, it follows the shape of the normal distribution.<p>Where it sucks for call centres is that demand shows seasonality and shift planning has relatively inflexible lead times. But in a software scenario we can typically acquire additional capacity quickly, so precision in forecasts is less of a problem.<p>A decent explanation: <a href="https:&#x2F;&#x2F;www.networkpages.nl&#x2F;the-golden-rule-of-staffing-in-contact-centers&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.networkpages.nl&#x2F;the-golden-rule-of-staffing-in-c...</a><p>A short scenario: <a href="https:&#x2F;&#x2F;www.xaprb.com&#x2F;blog&#x2F;square-root-staffing-law&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.xaprb.com&#x2F;blog&#x2F;square-root-staffing-law&#x2F;</a><p>A simple calculator linked from the short scenario: <a href="https:&#x2F;&#x2F;www.desmos.com&#x2F;calculator&#x2F;8lazp6txab" rel="nofollow">https:&#x2F;&#x2F;www.desmos.com&#x2F;calculator&#x2F;8lazp6txab</a>

If we were to extend the first graph a bit more to the right, the linear improvement would quickly trend downwards into negative latencies, a sure sign that it&#x27;s not the right answer. But if linear is impossible, then super-linear is just as impossible. The line the author describes as such is clearly asymptotic.<p>If we were discussing overall latency, and not just queue delay, then an asymptotic decrease towards zero could correspond to a super-linear increase in throughput capacity. But that&#x27;s not what&#x27;s happening here, either: because average latency is bounded below at one second, total throughput can never exceed <i>c</i>. There is no super-linearity to be found here.