Saving on egress switching from AWS to Hetzner

It&#x27;s the same story every day, the same pattern. And it&#x27;s the same because it works.<p>Startup: use &lt;aws|azure|gcp&gt; because it&#x27;s quick to get started. It&#x27;s easy to scale it up 100X in 5 minutes for when you make it big. And it provides a long list of easy to use services that are overpriced per use, but which free you to focus on your business rather than complex DevOps.<p>Then once you scale to the point that you&#x27;re comparing your salaries and your ops cost, you find smaller services to handle the expensive part of your business. Ones that do exactly what you need, not much else, and charge accordingly. Maybe they&#x27;re self-operated, or maybe they&#x27;re other companies with a narrow focus.<p>Then you write a blog post about how everyone is crazy to use &lt;cloud&gt; since it&#x27;s so expensive, and you don&#x27;t know why you ever thought they were a good idea.

For those in the Bay Area who are a bit more adventurous, Hurricane Electric charges only $400&#x2F;mo for a full rack in their Fremont datacenter, along with 1Gbps of bandwidth, and 12A of usable power.<p>It’s totally overkill for the article’s use case, but it enables many cool things that need sustained bandwidth, and has been pretty fun.

Beware that Hetzner is quick at blocking IPs, with zero recourse or explanation.<p>If they receive any kind of abuse report, they instantly put you on a 24h timer, but they won&#x27;t show you contents of that message. If you do not respond in time, they will simply block the IP. What&#x27;s strange is that I was using Cloudflare in front of Hetzner, but it seems that Cloudflare is ratting their users out and forwarding any abuse mail to the owners of &quot;cloud protected&quot; IPs, all without any notification or warning. Hetzner then sends you an email like &quot;Please remove &lt;sitename&gt; from our network within the next 24 hours. This site violates 6.2 of Hetzner&#x27;s terms and conditions.&quot;, where site in question is an obvious mirror&#x2F;search engine of another social network.<p>What&#x27;s perhaps more irritating is that Hetzner Cloud has a tiny LIFO queue of ipv4 addresses unique per customer, so you will get the same IP as soon as you release one. You can imagine what happens to a Rancher-managed k8s cluster with Hetzner node driver when one of those IPs gets a no-route: it will get stuck trying to re-deploy a failed node. Mine eventually fell apart though because I did not attend to it for a couple of years.<p>I&#x27;m now hosting from a proverbial garage (NixOS and cron) but proxying traffic through a cheap cloud VM, which saves me at least $1500&#x2F;mo.

Watch out that electricity costs went up significantly in Europe in the past months. I am a happy hetzner customer since 2012 I think. Their server monthly costs were just a bit higher than my electricity bill would&#x27;ve been if I were to keep a machine up 24&#x2F;7 at home, so I rented a machine from them having the hardware basically for free.<p>Last week I got an email from them saying my server cost would increase from 30eur&#x2F;month to 48eur&#x2F;month. I can take afford the increase, but I wonder how many companies with infra bills of tens or hundreds of thousands will be able to afford a 60% increase in hosting costs.

Great article. I wrote about a similar experience 9 months ago while switching from Google Cloud Storage to Scaleway Object Storage:<p><a href="https:&#x2F;&#x2F;pellegrino.link&#x2F;2021&#x2F;05&#x2F;31&#x2F;scaleway-object-storage-s3-java-library.html" rel="nofollow">https:&#x2F;&#x2F;pellegrino.link&#x2F;2021&#x2F;05&#x2F;31&#x2F;scaleway-object-storage-s...</a><p>Similarly, I saved about x100. For sure, the services do not offer the same level of guarantees and speed but depending on my case it was worth a migration.<p>Looking forward to trying Cloudflare R2 soon!

One thing Hetzner doesn’t have yet (but it SHOULD!) is Object Storage, and it’s something I’m working on over at NimbusWS[0].<p>Another awesome thing about Hetzner is that bandwidth internally is free and automatically negotiated (i.e if you send traffic to a Hetzner IP it will flow internally).<p>One of the hardest things is being able to tell the bandwidth usage and when it’s inside&#x2F;outside Hetzner.<p>[0]: <a href="https:&#x2F;&#x2F;nimbusws.com" rel="nofollow">https:&#x2F;&#x2F;nimbusws.com</a>

I work at Microsoft (not on Azure) and run Tor relays. My Tor Exit hosts are Psychz Networks, Terrahost, and OVH, along with a middle relay on CenturyLink Fiber.<p>Why? Azure egress bandwidth is too expensive for a Tor relay, even for a well-paid software engineer like me. I have ~1.5 Gbps Tor bandwidth. If I were to put that on Azure, I would be paying a mortgage worth of bandwidth fees. Same can apply to AWS and Google Cloud.<p>However, Big Clouds charge more for bandwidth because (a) they&#x27;re a leader and (b) they have paid peering deals with Big Telecom (e.g. AT&amp;T, Deutsche Telekom, Telefonica, etc.). Smaller hosts like Hetzner don&#x27;t peer with those ISPs and rely on &quot;transit&quot;, but that also means slightly worse performance to Big Telecom.<p>In short, big clouds can use their market power and advanced technology to command a premium, whereas smaller clouds have to win over customers with lower prices to make up.<p>Tor relays are ill-suited for Big Clouds, but then your 5000-node Kubernetes cluster is ill-suited for OVH or Hetzner.<p>I do use the free Azure credit for Tor Bridges, however.

&gt; This came out to about a 2x savings over AWS.<p>Is that a typical way to describe cost savings? I would have said 1&#x2F;2 the cost of AWS. It almost sounds like cloudflare would pay them money (e.g. you go to the store and see something marked 200% off, so you get paid to buy it, like when oil prices went negative).<p>Related: I&#x27;ve always found the 2x, 10x, etc notation confusing. Does a 2x improvement mean a 100% increase (e.g. 10 -&gt; 20) or a 200% increase (e.g. 10 -&gt; 30)? Would someone ever say a 1x improvement or a 0.5x improvement?

&gt; DIYing it does come with additional maintenance burden, but so far we’ve found this manageable.<p>Tune in next month for their article on how much fun manually scaling everything is.

I&#x27;m not familiar with Hetzner at all. But it seems they are running a single self managed VM to serve these updates globally, and then comparing that to the cost of egress on a globally distributed content delivery network? What am I missing here? This is not the same thing. Nevermind the fact that AWS is an ecosystem with hundreds of products working in tandem to deliver solutions not just EC2.

I feel it’s unwise to simply accept “additional maintenance burden” as a “free” cost unless it’s under an hour or two a month.

What&#x27;s interesting to me is that from my limited experience with cloud services it feels like the technical skills to set up cloud hostings and a good architecture to be just as much work as setting up dedicated servers or even vps.<p>Or maybe being 38 makes me really old in the way I approach things...

<a href="https:&#x2F;&#x2F;www.cloudflare.com&#x2F;bandwidth-alliance&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.cloudflare.com&#x2F;bandwidth-alliance&#x2F;</a><p>although I hate to shill for Cloudflare, as I believe they are evil, but anyway there you go

[Disclaimer: I&#x27;m Co-Founder and CEO of <a href="https:&#x2F;&#x2F;www.vantage.sh" rel="nofollow">https:&#x2F;&#x2F;www.vantage.sh</a> - a cloud cost platform]<p>On this topic as its relevant, at Vantage we launched an automated cost recommendation that keeps an eye on egress out of Cloudfront and S3 and recommends when it makes financial sense to switch to another provider (in the linked blog post, Cloudflare) <a href="https:&#x2F;&#x2F;www.vantage.sh&#x2F;blog&#x2F;cloudflare-specific-cost-recommendations-for-amazon-cloudfront-and-amazon-s3-egress" rel="nofollow">https:&#x2F;&#x2F;www.vantage.sh&#x2F;blog&#x2F;cloudflare-specific-cost-recomme...</a> -- this has been very popular with current customers&#x2F;users.<p>We&#x27;re actually in process of adding a bunch of new infrastructure and service providers and should include Hetzner in our list of recommended providers but thought I&#x27;d mention for anyone wanting an automated approach to knowing when you hit the financial tipping point, this is one option.

Hetzner has some strange pathing issues to the US, such that using an EU vpn almost always gets you much faster speeds.<p>I blame level3 as that’s where it seems to go pear-shaped.

Whenever I see posts like this, I&#x27;d like to point out that both is possible at the same time.<p>Full on self hosting your own cloud comes usually with a lot more services than &quot;the&quot; cloud does because thats usually why the hoster is considered &quot;small&quot;.<p>In Hetzners case you will for sure at least have to self host a decent RDBMS and it better be replicated, which means more maintenance and servers.<p>But you do not have to put all your eggs into one basket! There are more than enough ways to split the load and you can even add dedicated servers in the mix too!<p>It should not be &quot;yep we are on azure, or yup we are on hetzner&quot; but &quot;my job as an architect is to combine the two for maximum value&quot; ;)

&gt; Having heard of Hetzner’s low egress costs (20TB free + €1.19&#x2F;TB&#x2F;month), we investigated what it would take to run our own server.<p>My understanding was that they dropped the per-TB billing a few years ago and now it&#x27;s just uncapped?

A couple of talking points come to mind:<p>The linked blog post mentions October 2021, while in December 2021 AWS lowered pricing for CloudFront where the first terabyte out to internet is now completely free:<p><a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;about-aws&#x2F;whats-new&#x2F;2021&#x2F;11&#x2F;aws-price-reduction-data-transfers-internet&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;about-aws&#x2F;whats-new&#x2F;2021&#x2F;11&#x2F;aws-price...</a><p>For larger use cases most cloud providers have custom or private pricing options available. For example in CloudFront pricing page AWS publicly states that &quot;Custom discounted pricing is available for customers willing to commit to a minimum of 10 TB of data transfer per month for 12 months or longer.&quot;<p>I have noted that most of the blog posts like the one linked above talk about migrating workloads around for savings of tens or hundreds of dollars while focusing on low level IaaS (VMs), and completely ignore the benefits of leveraging cloud services, APIs and related software ecosystems instead of DIY.

I&#x27;d be most interested to hear about situations one step up from there, where Hetzner&#x27;s per-TB-per-month pricing doesn&#x27;t work out favorably and you want to serve content reasonably quickly anywhere in the world.<p>It seems like you could build out a CDN on OVH using 1-5Gbps unmetered+guaranteed bandwidth and place servers in the US, Europe, and a few PoPs in Asia for relatively cheap, then use GeoDNS for balancing traffic. But it seems like OVH&#x27;s pricing offerings have been shifting over the last year to remove the &quot;guaranteed bandwidth&quot; in favor of &quot;unmetered bandwidth&quot; (potentially throttled 50%) on more machine types.<p>It would still require monitoring and managing bandwidth saturation per host, and it&#x27;s unclear how much extra hassle OVH adds. But in theory it seems easier than setting up and managing colos in many countries?

It&#x27;s quite easy when you only need to replace EC2. Since last year Hetzner also has a Loadbalancer service that can be used with VM instances but also in Kubernetes. It&#x27;s not so easy to replace AWS RDS, though (or other services for that matter but RDS seems to be one of the more basic services).

Could someone explain what the « website assets » limitation of CloudFlare is about?

They did not mention Terraform in the article but it might be of interest for people considering a switch that there&#x27;s a Terraform provider for hcloud.

What’s the best way to use generic compute to create an Serverless functions equivalent? I feel that&#x27;s the best case for this sort of compute here - well, that and build jobs, heh.

tl;dr Cloudflare offered a savings over AWS CloudFront, but we saved multiple orders of magnitude DIYing it on Hetzner. So far ops have not been too painful.

This is interesting... Is there some distributed caching server one could use for this, for increased uptime? Basically some CDN.

I saw that the article mentioned maintenance being minimal right now but at some point, you will need to hire an SRE or some such individual to handle just the Hetzner subsystem for you. Is that part of the TCO calculations?<p>E.g., if you’re paying someone $5-7000&#x2F;month (no idea what an SRE costs) to manage a fleet of say 10 subsystems, their salary <i>should</i> factor into the cost calculations, right?

tl;dr the title was enough. I am so surprised by how this somewhat normal post (we got burned on aws cryptic billing model) makes it on the 1st place - is like people discovering warm water. This is not high-tech&#x2F;rocket technology&#x2F;science.

Bragging about switching to Hetzner always reminds me of my friend bragging about buying a $500 honda civic. What a great deal! He soon had a much larger tool chest and oil-stained jeans, and learned a lot more about cars :)