Reining in the thundering herd: Getting to 80% CPU utilization with Django

Tangent, but I always had a different understanding of the “thundering herd” problem; that is, if a service is down for whatever reason, and it’s brought back online, it immediately grinds to a halt again because there are a bazillion requests waiting to be handled.<p>And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the door immediately.

Unfortunately HAProxy doesn&#x27;t buffer requests*, which is necessary for a production deployment of gunicorn. And for anybody using AWS, ALB doesn&#x27;t buffer requests either. Because of this I&#x27;m actually running both HAProxy and nginx in front of my gunicorn instances—nginx in front for request buffering and HAProxy behind that for queuing.<p>If anybody is interested, I&#x27;ve packaged both as Docker containers:<p>HAProxy queuing&#x2F;load shedding: <a href="https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;luhn&#x2F;spillway" rel="nofollow">https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;luhn&#x2F;spillway</a><p>nginx request buffering: <a href="https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;luhn&#x2F;gunicorn-proxy" rel="nofollow">https:&#x2F;&#x2F;hub.docker.com&#x2F;r&#x2F;luhn&#x2F;gunicorn-proxy</a><p>* It does have an http_buffer_request option, but this only buffers the first 8kB (?) of the request.

This is somewhat suspect. At my place of work, we operate a rather large Python API deployment (over an order of magnitude more QPS than the OP&#x27;s post). However, our setup is... pretty simple. We only run nginx + gunicorn (gevent reactor), 1 master process + 1 worker per vCPU. In-front of that we have an envoy load-balancing tier that does p2c backend selection to each node. I actually think the nginx is pointless now that we&#x27;re using envoy, so that&#x27;ll probably go away soon.<p>Works amazingly well! We run our python API tier at 80% target CPU utilization.

HAProxy is a beautiful tool but it doesn&#x27;t buffer requests that is why NGINX is recommended in front of gunicorn otherwise it&#x27;s suspectible to slowloris attack. So either cloubhouse can be easily DDOS&#x27;d right now or they have some tricky setup that prevents slow post reqests reaching gunicorn. In the blog post they don&#x27;t mention that problem while recommend others to try and replace NGINX with HAPRoxy.

Performance is the only thing that is holding me back to consider Python for bigger web applications.<p>Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake.<p>Pypy could be a solution as it seems to be about 6x faster on average.<p>Is anybody here using Pypy for Django?<p>Did Clubhouse document somewhere if they are using CPython or Pypy?

&gt; Which exacerbated another problem: uWSGI is so confusing. It’s amazing software, no doubt, but it ships with dozens and dozens of options you can tweak.<p>I am glad I am not the only one. I&#x27;ve had so many issues with setting up sockets, both with gevent and uWSGI, only to be left even more confused after reading the documentation.

If you’re delegating your load balancing to something else further up the stack and would prefer a simpler WSGI server than Gunicorn, Waitress is worth a look: <a href="https:&#x2F;&#x2F;github.com&#x2F;pylons&#x2F;waitress" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pylons&#x2F;waitress</a>

Aside: AWS only allows registering 1000 targets in a target group… i wonder if thats the limit they hit. If so, its documented.

Have to wonder how well haproxy works vs balancing by making gunicorn listen via SO_REUSEPORT and letting the kernel balance instead (ala <a href="https:&#x2F;&#x2F;talawah.io&#x2F;blog&#x2F;extreme-http-performance-tuning-one-point-two-million&#x2F;" rel="nofollow">https:&#x2F;&#x2F;talawah.io&#x2F;blog&#x2F;extreme-http-performance-tuning-one-...</a>)

Interesting to read that they are using Unix sockets to send traffic to their backend processes. I know that it&#x27;s easily done when using HaProxy but I have never read about people using it. I guess the fact that they are not using docker or another container runtime makes sockets rather simple to use.

&gt; Python&#x27;s model of running N separate processes for your app is not as unreasonable as people might have you believe! You can achieve reasonable results this way, with a little digging.<p>I have been through this journey, we eventually migrated to Golang and it saved a ton of money and firefighting time. Unfortunately, python community hasnt been able to remove GIL, it has its benefits (especially for single threaded programs), but I believe the cost (lack of concurrent abstractions. async&#x2F;await doesn&#x27;t cut it) far outweigh it.<p>Apart from what the article mentions, other low hanging fruits worth exploring are<p>[1] Moving under PyPy (this should give some perf for free)<p>[2] Bifurcate metadata and streaming if not already. All the django CRUD stuff could be one service, but the actual streaming should be separated to another service altogether.

Interesting to see this. It sounds like they&#x27;re not on AWS, given that they mentioned that having 1000 instances for their production environment made them one of the bigger deployments on their hosting provider.<p>If not for the troubles they experienced with their hosting provider and managing deployments &#x2F; cutting over traffic, it possibly could have been the cheaper option to just keep horizontally scaling vs putting in the time to investigate these issues. I&#x27;d also love to see some actual latency graphs, what&#x27;s the P90 like at 25% CPU usage with a simple Gunicorn &#x2F; gevent setup?

Kinda funny they decided paying a ton of money to aws was ok but paying for nginx plus was not

ya, I pretty much agree with 3 suggestions at the end:<p>* use uWSGI (read the docs, so many options...)<p>* use HAProxy, so very very good<p>* scale python apps by using processes.

If it is just a backend, why not port it over to one of the myriad of cloud autoscaling solutions that are out there?<p>The opportunity cost of spending time figuring out why only 29 workers are receiving requests over adding new features that generate more revenue, seems like a quick decision.<p>Personally, I just start off with that now in the first place, the development load isn&#x27;t any greater and the solutions that are out there are quite good.

I&#x27;ve always used nginx for my servers. Is HAProxy that much better to consider learning&#x2F;switching?

1M requests per minute on 1000 web instances is not an achievement, it is a disaster.<p>It is ridiculous people brag about it.<p>Guys, if you have budget maybe I can help you up this by couple orders of magnitude.

use PHP ;)

Famous last words, but I get the sense that the need to handle this sort of load on Clubhouse is plateauing and will decline from here. The app seems to have shed all the people that drew other people initially and lost its small, intimate feel and has turned into either crowded rooms where no one can say anything, or hyper specific rooms where no one has anything to say.<p>Good article though! I’ve dealt with these exact issues and they can be very frustrating.

I wouldn&#x27;t be very proud of writing an article like that.<p>Usually engineering blogs exists to show that there are fun stuff to do in a company. But here it just seems they have no idea, what they are doing. Which is fine, I&#x27;m classifying myself in the same category.<p>Reading the article I don&#x27;t feel like they have solved their issue, they just created more future problems