GitHub: Recent Load Balancer Problems (RCA)

Reminds me of the last Github post I'd read about their load-balancer setup from their server guy(2009): <a href="http://www.anchor.com.au/blog/2009/10/load-balancing-at-github-why-ldirectord/" rel="nofollow">http://www.anchor.com.au/blog/2009/10/load-balancing-at-gith...</a><p>Willy Tarreau (author of HAproxy) had sparked a nice discussion in the comments section, that time.

The post mentions that heartbeats timeout when the load spikes momentarily. I have a few questions, would love to hear answers if it's okay to share :-)<p><pre><code> 1. What load spiked? Is it the network/CPU load? 2. By spiked (be it network or CPU), do you mean the load went all the way to 100%? Or was it some threshold like say 90% of the available capacity? 3. What's the heartbeat time interval? </code></pre> Thanks, (EDIT: spacing)

They actually have a pretty good HA setup.<p>One thing to remember is that a HA cluster is for handling node failure (power loss, faulty hardware, faulty software, etc). It is not for handling capacity related failure. If the servers are overloaded with too many requests, they will fail regardless of the HA setup. Capacity monitoring and capacity planning are still needed to maintain uptime.

I rarely see an install of Heartbeat/Pacemaker/CRM preventing more downtime than they cause. If you add in DRBD on top, you get an entire suite of false-HA infrastructure.