Zero-Downtime Kubernetes Deployments on AWS with EKS

We had to figure this out the hard way, and ended up with this approach (approximately).<p>K8S provides two (well three, now) health checks.<p>How this interacts with ALB is quite important.<p>Liveness should always return 200 OK unless you have hit some fatal condition where your container considers itself dead and wants to be restarted.<p>Readiness should only return 200 OK if you are ready to serve traffic.<p>We configure the ALB to only point to the readiness check.<p>So our application lifecycle looks like this:<p>* Container starts<p>* Application loads<p>* Liveness begins serving 200<p>* Some internal health checks run and set readiness state to True<p>* Readiness checks now return 200<p>* ALB checks begin passing and so pod is added to the target group<p>* Pod starts getting traffic.<p>time passes. Eventually for some reason the pod needs to shut down.<p>* Kube calls the preStop hook<p>* PreStop sends SIGUSR1 to app and waits for N seconds.<p>* App handler for SIGUSR1 tells readiness hook to start failing.<p>* ALB health checks begin failing, and no new requests should be sent.<p>* ALB takes the pod out of the target group.<p>* PreStop hook finishes waiting and returns<p>* Kube sends SIGTERM<p>* App wraps up any remaining in-flight requests and shuts down.<p>This allows the app to do graceful shut down, and ensures the ALB doesn&#x27;t send traffic to a pod that knows it is being shut down.<p>Oh, and on the Readiness check - your app can use this to (temporarily) signal that it is too busy to serve more traffic. Handy as another signal you can monitor for scaling.<p>e: Formatting was slightly broken.

I know this won&#x27;t be helpful to folks committed to EKS, but AWS ECS (i.e. running docker containers with AWS controlling) does a really great job on this, we&#x27;ve been running ECS for years (at multiple companies), and basically no hiccups.<p>One of my former co-workers went to a K8S shop, and longs for the simplicity of ECS.<p>No software is a panacea, but ECS seems to be one of those &quot;it just works&quot; technologies.

I run <a href="https:&#x2F;&#x2F;BareMetalSavings.com" rel="nofollow">https:&#x2F;&#x2F;BareMetalSavings.com</a>.<p>The amount of companies who use K8s when they have no business nor technological justification for it is staggering. It is the number one blocker in moving to bare metal&#x2F;on prem when costs become too much.<p>Yes, on prem has its gotchas just like the EKS deployment described in the post, but everything is so much simpler and straightforward it&#x27;s much easier to grasp the on prem side of things.

I&#x27;m not sure why they state &quot;although the AWS Load Balancer Controller is a fantastic piece of software, it is surprisingly tricky to roll out releases without downtime.&quot;<p>The AWS Load Balancer Controller uses readiness gates by default, exactly as described in the article. Am I missing something?<p>Edit: Ah, it&#x27;s not by default, it requires a label in the namespace. I&#x27;d forgotten about this. To be fair though, the AWS docs tell you to add this label.

A few years ago, while helping build a platform on Google Cloud &amp; GKE for a client, we found the same issues.<p>At that point we already had a CRD used by most of out tenant apps, which deployed an opinionated (but generally flexible enough) full app stack (Deployment, Service, PodMonitor, many sane defaults for affinity&#x2F;anti-affinity, etc, lots of which configurable, and other things).<p>Because we didn&#x27;t have an opinion on what tenant apps would use in their containers, we needed a way to make the pre-stop sleep small but OS-agnostic.<p>We ended up with a 1 LOC (plus headers) C app that compiled to a tiny static binary. This was put in a ConfigMap, which the controller mounted on the Pod, from where it could be executed natively.<p>Perhaps not the most elegant solution, but a simple enough one that got the job done and was left alone with zero required maintenance for years - it might still be there to this day. It was quite fun to watch the reaction of new platform engineers the first time they&#x27;d come across it in the codebase. :D

This is actually a fascinatingly complex problem. Some notes about the article: * The 20s delay before shutdown is called “lame duck mode.” As implemented it’s close to good, but not perfect. * When in lame duck mode you should fail the pod’s health check. That way you don’t rely on the ALB controller to remove your pod. Your pod is still serving other requests, but gracefully asking everyone to forget about it. * Make an effort to close http keep-alive connections. This is more important if you’re running another proxy that won’t listen to the health checks above (eg AWS -&gt; Node -&gt; kube-proxy -&gt; pod). Note that you can only do that when a request comes in - but it’s as simple as a Connection: close header on the response. * On a fun note, the new-ish kubernetes graceful node shutdown feature won’t remove your pod readiness when shutting down.

&gt;The truth is that although the AWS Load Balancer Controller is a fantastic piece &gt;of software, it is surprisingly tricky to roll out releases without downtime.<p>20 years ago we used simple bash scripts using curl to do rest calls to take one host out of our load balancers, then scp to the host and shut down the app gracefully, and updated the app using scp again, then put it back into the load balancer after testing the host on its own. we had 4 or 5 scripts max, straightforward stuff..<p>They charge $$$ and you get downtime in this simple scenario ?

The fact that the state of the art container orchestration system requires you to run a sleep command in order to not drop traffic on the floor is a travesty of system design.<p>We had perfectly good rolling deploys before k8s came on the scene, but k8s insistence on a single-phase deployment process means we end up with this silly workaround.<p>I yelled into the void about this once and I was told that this was inevitable because it&#x27;s an eventually consistent distributed system. I&#x27;m pretty sure it could still have had a 2 phase pod shutdown by encoding a timeout on the first stage. Sure, it would have made some internals require more complex state - but isn&#x27;t that the point of k8s? Instead everyone has to rediscover the sleep hack over and over again.

Nit: &quot;How we archived&quot; subheading should be &quot;How we achieved&quot;.

We’re using Argo rollouts without issue. It’s a super set of a deployment with configuration based blue green deploy or canary. Works great for us and allows us to get around the problem laid out in this article.

Does this or any of the strategies listed in the comments properly handle long lived client connections? It&#x27;s sufficient enough to wait for the LB to stop sending traffic when connections are 100s of ms or less but when connections are minutes or even hours long it doesn&#x27;t work out well.<p>Is there a slick strategy for this? Is it possible to have minutes long pre-stop hooks? Is the only option to give client connections an abandon ship message and kick them out hopefully fast enough?

Might be noteworthy that in recent enough k8s lifecycle.preStop.sleep.seconds is implemented <a href="https:&#x2F;&#x2F;github.com&#x2F;kubernetes&#x2F;enhancements&#x2F;blob&#x2F;master&#x2F;keps&#x2F;sig-node&#x2F;3960-pod-lifecycle-sleep-action&#x2F;README.md#summary" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kubernetes&#x2F;enhancements&#x2F;blob&#x2F;master&#x2F;keps&#x2F;...</a> so no longer any need to run an external sleep command.

highly recommend porter if you are a startup who doesn&#x27;t wanna think about things like this

somewhat related <a href="https:&#x2F;&#x2F;architect.run&#x2F;" rel="nofollow">https:&#x2F;&#x2F;architect.run&#x2F;</a><p>&gt; Seamless Migrations with Zero Downtime<p>(I don&#x27;t work for them but they are friends ;))