The blog post announcing this might be a better read:<p><a href="https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581" rel="nofollow">https://medium.com/@NetflixTechBlog/performance-under-load-3...</a><p>(And while reading that and thinking "I swear I recently read something else talking about applying TCP congestion control to RPC queueing". And indeed I had: <a href="http://www.evanjones.ca/prevent-server-overload.html" rel="nofollow">http://www.evanjones.ca/prevent-server-overload.html</a>)
This reminds me a lot of the sbroker library in Erlang:<p>> SBroker is a framework to build fault tolerant task schedulers. It includes several built in schedulers based on TCP (and other network scheduling) algorithms.<p>> [...] in an ideal situation a target queue time would be chosen that keeps the system feeling responsive and clients would give up at a rate such that in the long term clients spend up to the target time in the queue. This is sojourn (queue waiting) time active queue management. CoDel and PIE are two state of the art active queue management algorithms with a target sojourn time, so should use those with defaults that keep systems feeling responsive to a user.<p><a href="http://www.erlang-factory.com/euc2017/james-fish" rel="nofollow">http://www.erlang-factory.com/euc2017/james-fish</a><p><a href="https://github.com/fishcakez/sbroker" rel="nofollow">https://github.com/fishcakez/sbroker</a>
I like the sounds of this:<p>"<i>Executor -- The BlockingAdaptiveExecutor adapts the size of an internal thread pool to match the concurrency limit based on measured latencies of Runnable commands and will block when the limit has been reached.</i>"<p>I'm often surprised this kind of auto-scaling thread pool is not a more common thing in Java land.
PROTIP: Managing congestion control on application level is not a good business. A better idea is to leave it to the edge and CDN, while the app level uses computationally cheap optimistic algos since the com in between the app and your edge will go over your own high quality infrastructure.<p>This adds flexibility to allow for use of multiple algorithms in different load balancing regions (mobile as a lossy fabric is better to stay with conventional, desktop and server clients can use a some smarter throughput maximizing algo, and in countries with high percentages of connections being laggy DSLs, you can use something else )
This library looks very interesting. I've used a similar approach for pulling batches of items from a queue (e.g. discover the optimal batch size and inter-poll wait time). There are plenty of other places we could benefit from something like this. I can't wait to try this out.<p>Netflix puts out some amazing Java libraries. I've had excellent results using Hystrix [0]. It has been an excellent addition to our systems.<p>[0] <a href="https://github.com/Netflix/Hystrix" rel="nofollow">https://github.com/Netflix/Hystrix</a>
This is a pretty cool design if your requests to a given endpoint are supposed to all take about the same time. It's not easy to see how you'd adjust it for things with more variance; perhaps rather than using the fastest seen mtt, you could look at your p99 over the last N minutes, and see if it's been changing?
Would this work when connecting multiple threadpools (java executors)? Imagine I have a microservice where first threadpool downloads large files to disk (IO bound) and another threadpool that processes the downloaded data (CPU bound). Those two threadpools communicate with a bounded queue. Will using the concurrency-limit Executor allow me to get the best throughput in this scenario?
Can anyone explain how they decide which requests to reject? The blog post just mentions that excess RPS gets rejected, but couldn't rejecting arbitrary requests cause other problems?