Kubernetes on Hetzner: cutting my infra bill by 75%

We&#x27;ve [1] been using Hetzner&#x27;s dedicated servers to provide Kubernetes clusters to our clients for a few years now. The performance is certainly excellent, we typically see request times half. And because the hardware is cheaper we can provide dedicated DevOps engineering time to each client. There are some caveats though:<p>1) A staging cluster for testing updates is really a must. YOLO-ing prod updates on a Sunday is no one&#x27;s idea of fun.<p>2) Application level replication is king, followed by block-level replication (we use OpenEBS&#x2F;Mayastor). After going through all the Postgres operators we found StackGres to (currently) be the best.<p>3) The Ansible playbooks are your assets. Once you have them down and well-commented for a given service then re-deploying that service in other cases (or again in the future) becomes straightforward.<p>4) If you can I&#x27;d recommend a dedicated 10G network to connect your servers. 1G just isn&#x27;t quite enough when it comes to the combined load of prod traffic, plus image pulls, plus inter-service traffic. This also gives a 10x latency improvement over AWS intra-az.<p>5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird).<p>6) MinIO clusters (via the operator) are not that tricky to operate as long as you follow the well trodden path. This provides you with local high-bandwidth, low-latency object storage.<p>7) The initial investment to do this does take time. I&#x27;d put it at 2-4 months of undistracted skilled engineering time.<p>8) You can still push ancillary&#x2F;annoying tasks off onto cloud providers (personally I&#x27;m a fan of CloudFlare for HTTP load balancing).<p>[1]: <a href="https:&#x2F;&#x2F;lithus.eu" rel="nofollow">https:&#x2F;&#x2F;lithus.eu</a>

I have experience running Kubernetes clusters on Hetzner dedicated servers, as well as working with a range of fully or highly managed services like Aurora, S3, and ECS Fargate.<p>From my experience, the cloud bill on Hetzner can sometimes be as low as 20% of an equivalent AWS bill. However, this cost advantage comes with significant trade-offs.<p>On Kubernetes with Hetzner, we managed a Ceph cluster using NVMe storage, MariaDB operators, Cilium for networking, and ArgoCD for deploying Helm charts. We had to handle Kubernetes cluster updates ourselves, which included facing a complete cluster failure at one point. We also encountered various bugs in both Kubernetes and Ceph, many of which were documented in GitHub issues and Ceph trackers. The list of tasks to manage and monitor was endless. Depending on the number of workloads and the overall complexity of the environment, maintaining such a setup can quickly become a full-time job for a DevOps team.<p>In contrast, using AWS or other major cloud providers allows for a more hands-off setup. With managed services, maintenance often requires significantly less effort, reducing the operational burden on your team.<p>In essence, with AWS, your DevOps workload is reduced by a significant factor, while on Hetzner, your cloud bill is significantly lower.<p>Determining which option is more cost-effective requires a thorough TCO (Total Cost of Ownership) analysis. While Hetzner may seem cheaper upfront, the additional hours required for DevOps work can offset those savings.

&gt; Hetzner volumes are, in my experience, too slow for a production database. While you may in the past have had a good experience running customer-facing databases on AWS EBS, with Hetzner&#x27;s volumes we were seeing &gt;50ms of IOWAIT with very low IOPS.<p>There is a surprisingly easy way to address this issue: use (ridiculously cheap) Hetzner metal machines as nodes. The ones with nvme storage offer excellent performance for dbs and often have generous amounts of RAM. I&#x27;d go as far as to say you&#x27;d be better off to invest in two or more beefy bare metal machines for a master-replica(s) setup rather than run the db on k8s.<p>If you don&#x27;t want to be bothered with the setup, you can use one of many modern packages such as Pigsty: <a href="https:&#x2F;&#x2F;pigsty.cc&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pigsty.cc&#x2F;</a> (not affiliated but a huge fan).

Been a happy Hetzner customer for over a decade, previously using their dedicated servers in their German DC&#x27;s before migrating to their Cloud US VMs for better latency with the US. Slightly disappointed with their recent cuts of their generous 20TB free traffic down to 3TB (€1.19 per additional TB), but they still look to be a lot better value than all other US cloud providers we&#x27;ve evaluated.<p>Whilst I wouldn&#x27;t run Kubernetes by choice, we&#x27;ve had success moving our custom SSH &#x2F; Docker compose deployments over to use GitHub Actions with kamal-deploy.org, easy to setup and nice UX tools for monitoring remote deployed apps [1]<p>[1] <a href="https:&#x2F;&#x2F;servicestack.net&#x2F;posts&#x2F;kamal-deployments" rel="nofollow">https:&#x2F;&#x2F;servicestack.net&#x2F;posts&#x2F;kamal-deployments</a>

I used to do my own car maintenance, because I wanted to save money, and it was fun. It turned out it was more complex than I thought, things slowly fell apart or broke. I spent a good deal of time &quot;re-fixing&quot; things. Spent probably thousands on tools over the years, partly replacing the cheap stuff that broke or rusted quickly. My cars were often up on blocks. But I learned a lot of great lessons. The biggest one? Some things are not worth DIYing; pay a mechanic or lease your car, especially if you depend on it for your livelihood.<p>Even something as simple as an oil change, really isn&#x27;t worth doing yourself. First you buy the tools (oil drip pan, filter wrench, funnel, creeper). Then you set aside the time to use them, find your dingy work clothes. You go to the store and buy new oil and a filter. You go home and change the oil. Then that day or another day you go to a store that will take your used oil. Versus 20 minutes at an auto mechanic, for about $15 more than the cost of the oil and filter.<p>Kubernetes is an entire car (and a complex one). It&#x27;s <i>really</i> not worth doing the maintenance yourself, I promise you. Unless you&#x27;re just doing it for fun.

This is an interesting writeup, but I feel like it&#x27;s missing a description of the cluster and the workload that&#x27;s running on it.<p>How many nodes are there, how much traffic does it receive, what are the uptime and latency requirements?<p>And what&#x27;s the absolute cost savings? Saving 75% of $100K&#x2F;mo is very different from saving 75% of $100&#x2F;mo.

When I worked in web hosting (more than 10 years ago), we would constantly be blackholeing Hetzner IPs due to bad behavior. Same with every other budget&#x2F;cheap vm provider. For us, it had nothing to do with geo databases, just behavior.<p>You get what you pay for, and all that.

I work for a consultancy company that helps companies building and securing infrastructure. We have a lot of customers running Kubernetes at low-cost providers (like Hetzner), more local middle-tier and top-three (AWS, GCP, Azure). We also have some governmental, financial and medical companies that can not or will not run in public clouds, so they usually host on-prem.<p>If Hetzner has an issue or glitch once a month, the middle-tier providers have one every 2-3 months, and a place like AWS maybe every 5-6 months. However, prices also follow that observation, so you have to carefully consider on a case-by-case basis whether adding some extra machines and backup and failure scenarios is a better deal.<p>The major benefit by using basic hosting services is that their pricing is a lot more predictable; you pay for machines and scale as you go. Once you get hooked into all the extra services a provider like AWS provides, you might get some unexpectedly high bills and moving away might be a lot harder. For smaller companies, don&#x27;t make short-sighted decisions that threaten your ability to survive long-term by choosing the easy solution or &quot;free credits&quot; scheme early on.<p>There is no right answer here, just trade-offs.

I haven&#x27;t used it personally, but <a href="https:&#x2F;&#x2F;github.com&#x2F;kube-hetzner&#x2F;terraform-hcloud-kube-hetzner">https:&#x2F;&#x2F;github.com&#x2F;kube-hetzner&#x2F;terraform-hcloud-kube-hetzne...</a> looks amazing as a way to setup and manage kubernetes on Hetzner. At the moment I&#x27;m on Oracle free tier, but I keep thinking about switching to it to get off... Well Oracle.

&gt; While DigitalOcean, like other providers, offers a free managed control plane, there is typically a 100% markup on the nodes that belong to these managed clusters.<p>I don&#x27;t think this is true. With Digital Ocean, the worker nodes are the same cost as regular droplets, there&#x27;s no additional costs involved. This makes Digital Ocean&#x27;s offering very attractive - free control plane you don&#x27;t have to worry about, free upgrades, and some extra integrations to things like the load balancer, storage, etc. I can&#x27;t think of a reason to not go with that over self-managed.

I loved the article. Insightful, and packed with real world applications. What a gem.<p>I have a side-question pertaining to cost-cutting with Kubernetes. I&#x27;ve been musing over the idea of setting up Kubernetes clusters similar to these ones but mixing on-premises nodes with nodes from the cloud provider. The setup would be something like:<p>- vCPUs for bursty workloads,<p>- bare metal nodes for the performance-oriented workloads required as base-loads,<p>- on-premises nodes for spiky performance-oriented workloads, and dirt-cheap on-demand scaling.<p>What I believe will be the primary unknown is egress costs.<p>Has anyone ever toyed around with the idea?

The key take home point here is not how amazingly cheap Hetzner is, which it is. But how much of an extortion game Google, Amazon, MS, etc. are playing with their cloud services. These are trillion dollar companies because they are raking in cash with extreme margins.<p>Yes, there is some added value in the level of convenience provided. But maybe with a bit more competition, pricing could be more competitive. A lot more competitive.

&gt; Hetzner volumes are, in my experience, too slow for a production database. While you may in the past have had a good experience running customer-facing databases on AWS EBS, with Hetzner&#x27;s volumes we were seeing &gt;50ms of IOWAIT with very low IOPS. See <a href="https:&#x2F;&#x2F;github.com&#x2F;rook&#x2F;rook&#x2F;issues&#x2F;14999">https:&#x2F;&#x2F;github.com&#x2F;rook&#x2F;rook&#x2F;issues&#x2F;14999</a> for benchmarks.<p>I set up rook ceph on a talos k8s cluster (with vm volumes) and experienced similar low performance; however, I always thought that was because of the 1Gi vSwitch (i.e. networking problem)?! The SSD volumes were quite fast.

Be careful with Hetzner, they null routed my game server on launch day due to false positives from their abuse system, and then took 3 days for their support team to re-enable traffic.<p>By that point I had already moved to a different provider of course.

As far as I see, no one is mentioning sustainability AKA environmental impact or &#x27;green hosting&#x27; here. Don&#x27;t you care about that?<p>I believe that Hetzner data centers in Europe (Germany, Finland) are powered by green energy, but not the locations in US.

Hi Bill, Wow! Thanks for the amazing write-up and for sharing it on your blog and here! I am so happy that we&#x27;ve helped you save so much money and that you&#x27;re happy with our support team! It&#x27;s a great way to start off the week! --Katie

Can anybody speak to the pros and cons of Hetzner vs OVH?<p>There ain&#x27;t many large European cloud companies, and I would like to understand how they differentiate.<p>Ionos is another European one. Currently, it looks like their cloud business is stagnating, though.

This is probably out of left field, but what is the benefit of having a naming scheme for nodes without any delimiters? Reading at a glance and not knowing the region name convention of a given provider (i.e. Hetzner), I&#x27;m at a loss to quickly decipher the &quot;&lt;region&gt;&lt;zone&gt;&lt;environment&gt;&lt;role&gt;&lt;number&gt;&quot; to &quot;euc1pmgr1&quot;. I feel like I&#x27;m missing something because having delimiters would make all sorts of automated parsing much easier.

That&#x27;s a really good article. Actually, recently we were migrating as well and we were using dedicated nodes in our setup.<p>In order to integrate a load-balancer provided by hetzner with our k8s on dedicated servers we had to implement a super thin operator that does it: <a href="https:&#x2F;&#x2F;github.com&#x2F;Intreecom&#x2F;robotlb">https:&#x2F;&#x2F;github.com&#x2F;Intreecom&#x2F;robotlb</a><p>If anyone will be inspired by this article and would want to do the same, feel free to use this project.

I’m planning on doing something similar but want to use Talos with bare metal machines. I suspect to see similar price reductions from our current EKS bill.

I went hetzner baremetal, set up a proxmox cluster over it and then have kubernetes on top. Gives me a lot of flexibility I find.

We&#x27;re very happy to use Hetzner for our bare-metal staging environments to validate functionality, but I still feel reluctant to put our production there. Disks don&#x27;t quite work as intended at all times and our vSwitch setup has gotten reset more than once.<p>All of this makes sense considering the extremely low price.

Very nicely written article. I’m also running a k8s cluster but on bare metal and qemu-kvms for the base load. Wonder why you would chose VMs instead of bare metal if you looking for cost optimisation (additional overhead maybe?), could you share more about this or did I miss it?

I feel lots of the work described in the article can be automated by kops, probably in a much better way, especially when it comes to day 2 operations.<p>I wonder what is the motivation behind manually spinning up a cluster instead of going with more established tooling?

We at Syself.com also have great experiences with Kubernetes on Hetzner. We built a platform on top of Cluster API and brought a managed Kubernetes experience to Hetzner. Now we have self-healing, automated updates and 100% reproductibility, with full bare metal support.<p>&gt; Hetzner volumes are, in my experience, too slow for a production database.<p>That&#x27;s true, though. To solve that we developed a way to persist the local storage of bare metal servers across reprovisionings. This way it&#x27;s both faster and cheaper. Now we are adding an automated database deployment layer on top of it.

<a href="https:&#x2F;&#x2F;github.com&#x2F;puppetlabs&#x2F;puppetlabs-kubernetes">https:&#x2F;&#x2F;github.com&#x2F;puppetlabs&#x2F;puppetlabs-kubernetes</a><p>What do the fine people of HN think about the size&#x2F;scope&#x2F;amount of technology of this repo?<p>It is referenced in the article here: <a href="https:&#x2F;&#x2F;github.com&#x2F;puppetlabs&#x2F;puppetlabs-kubernetes&#x2F;compare&#x2F;main...bilbof:puppetlabs-kubernetes:main#diff-50ae7fb3724b662b58dbc1c71663cb16a484ab36aecd5a11317fb14465f847fa">https:&#x2F;&#x2F;github.com&#x2F;puppetlabs&#x2F;puppetlabs-kubernetes&#x2F;compare&#x2F;...</a>

Funnily enough, we made the exact same transition from heroku to DigitalOceans managed Kubernetes service, and saved about 75%. Presumably this means that had we moved from heroku to hetzner, it would have been 93% savings!<p>The costs of cloud hosting are totally out of control, would love to see more efforts that lets developers move down the stack.<p>I’ve been humbly working on <a href="https:&#x2F;&#x2F;canine.sh" rel="nofollow">https:&#x2F;&#x2F;canine.sh</a> which basically provides a Heroku like interface to any K8 cluster

When I first started hosting servers&#x2F;services for customers I was using EC2 and Rackspace, then I discovered Linode and was happy it was so much cheaper with apparently no downside. After the first couple of interactions with support I started to relax. Then I discovered OVH, same story. I haven&#x27;t needed the support yet though.

&#x2F;&#x2F; Taking another slant at the discussion: Why kubernetes?<p>Thank you for sharing your experience. I also have my 3 personal servers with Hetzner, plus a couple VM instances in Scaleways (French outfit).<p>Disclaimer: I’m a Googler, was SRE for ~10 years for GMail, identity, social, apps (gsuites nowadays) and more, managed hundreds of jobs in Borg, one of the 3 founders of the current dev+devops internal platform (and I focused on the releases,prod,capacity side of the platform), dabbled in K8s on my personal time. My opinions, not Google’s.<p>So, my question is: given the significant complexity that K8s brings (I don’t think anyone disputes this) why are people using it outside medium-large environments? There are simpler and yet flexible &amp; effective job schedulers that are way easier to manage. Nomad is an example.<p>Unless you have a LOT of machines to manage, with many jobs (I’d say +250) to manage, K8s complexity, brittleness and overhead are not justifiable, IMO.<p>The emergence of tools like Terraform and the <i>many</i> other management layers in top of K8s that try to make it easier but just introduce more complexity and their own abstractions are in itself a sign of that inherent complexity.<p>I would say that only a few companies in the world need that level of complexity. And then they <i>will</i> need it, for sure. But, for most is like buying a Formula 1 to commute in a city.<p>One other aspect that I also noticed is that technical teams tend to carry on the mess they had in their previous “legacy” environment and just replicate in K8s, instead of trying to do an architectural design of the whole system needs. And K8s model enables that kind of mess: a “bucket of things”.<p>Those two things combined, mean that nowadays every company has soaring cloud costs, are running things they know nothing about but are afraid to touch in case of breaking something. And an outage is more career harming than a high bill that Finance will deal with it later, so why risk it, right? A whole new IT area has been coined now to deal with this: FinOps :facepalm:<p>I’m just puzzled by the whole situation, tbh.

Anybody running k3s&#x2F;k8s on Hetzner using cax servers? How&#x27;s that working?

For those considering Hetzner, there is also Contabo which is another German hosting company that is also good, at least in my experience

Did you try Cloud66 for deploy?

What about cluster autoscaling?

cut my kube bill 100% on GKE by switching from regional to zonal cluster bc the first zonal cluster is free

Do you know that they are cutting their free tier bandwidth? Did not read too much into it, but heard a few friends were worried about.<p>End of they day, they are a business!

Great write up Bill!

this is good<p>well, running on bare metal would be even better

Lovely website.