They were paying on-demand ec2 prices and reserved instances alone would save them ~35%, savings plan even more which would apply to egress and storage costs too. Anyway, they're still saving a lot more (~55%), but it's not nearly as egregious of a difference.
A lot of comments here seem to be along the lines of "you can hire one more engineer," but given the current economic situation remember that might be "keep one more engineer." Would you lay off someone on your team to keep AWS?<p>Keeping a few racks of servers happily humming along isn't the massive undertaking that most people here seem to think it is. I think lots of "cloud native" engineers are just intimidated by having to learn lower levels to keep things running.
If you are not using cloud services just virtual machines that's rarely worth it. And for a lot of people renting dedicated servers is the best business decision. It can be so much cheaper than EC2 that you solve your normal scalability problem by simply having a large amount of excess capacity. As you grow, some dedicated providers will give you an API to spin up a server in <120 seconds so even adding capacity quickly becomes possible. If your load is extremely spikey then this is not going to work but most people doesn't deal with loads like that -- or have other mitigation strategies.<p>Always keep an eye on your business goals and not on the hype. For oneuptime obviously downtime is a huge problem but you'd be surprised for how many businesses it's much cheaper to be down for a few minutes here and there than engineering a complex HA mechanism. The aforementioned spiking problem often can be solved cheaply by degrading the hot pages to static and serving them from CDN (if you have a mechanism for doing this of course). And so forth.<p>Remember KISS.
Moving from buying Ferraris to Toyota Camrys would save a lot of money too. These stories are always bs blog spam by companies trying to pretend they pulled off some amazing new hack. In reality they were burning cash because they hadn't the faintest idea how to control their spend.<p><pre><code> When we were utilizing AWS, our setup consisted of a 28-node managed Kubernetes cluster.
Each of these nodes was an m7a EC2 instance. With block storage and network fees included,
our monthly bills amounted to $38,000+
</code></pre>
The hell were you doing with 28 nodes to run an uptime tracking app? Did you try just running it on like, 3 nodes, without K8s?<p><pre><code> When compared to our previous AWS costs, we’re saving over $230,000 roughly per year
if you amortize the cap-ex costs of the server over 5 years.
</code></pre>
Compared to a 5-year AWS savings plan? Probably not.<p>On top of this, they somehow advertise using K8s as a simplification? Let's reign in our spend, not only by abandoning the convenience of VMs and having to do more maintenance, but let's require customers use a minimum of 3 nodes and a dozen services to run a dinky <i>uptime tracking app</i>.<p>This meme must be repeating itself due to ignorance. The CIOs/CTOs have no clue how to control spend in the cloud, so they rake up huge bills and ignore it "because we're trying to grow quickly!" Then maybe they hire someone who knows the Cloud, but they tell them to ignore the cost too. Finally they run out of cash because they weren't watching the billing, so they do the only thing they are technically competent enough to do: set up some computers and install Linux, and write off the cost as cap-ex. Finally they write a blog post in order to try to gain political cover for why they burned through several headcount worth of funding on nothing.
Weren't you searching for a colo provider just yesterday? That was a quick $230k! <a href="https://news.ycombinator.com/item?id=38275614">https://news.ycombinator.com/item?id=38275614</a>
So, they essentially saved the cost of one good engineer. Question is, are they spending 1 man-year of effort to maintain this setup themselves? If not, they made the right choice. Otherwise, it’s not as clear cut.
The post is super light on details, it's hard to visualize if it's worth it or not.<p>For examples:<p>- How much data are they working with? What's the traffic shape? Using NFS makes me think that they don't have a lot of data.<p>- What happened when their customers accidentally sent too much events? Will they simply drop the payload? In bare-metal they lose the ability to auto-scale quickly.<p>- Are they using S3 or not, if they are, did they move that as well to their own Ceph cluster?<p>- What's the RDBMS setup? Are they running their own DB proxy that can handle live switch-over and seamless upgrade?<p>- What's the details on the bare metal setup? Is everything redundant? How quickly can they add several racks in one go? What's included as a service from their co-lo provider?
We're in the business of making money. Our product makes the money. Not our skills and abilities to manage hardware and IT infrastructure.<p>They have a decent business case, but I don't feel like they executed well to meet the real objective. They don't want to be in AWS since they're an uptime monitor and they want to alert on downtime on AWS. But they have a single rack of servers in a single location. A cold standby in AWS doesn't mean a ton unless they're testing their failovers there... which comes at quite the cost.<p>I've worked on-prem before. Now I work somewhere that's 100% AWS and cloud native. You'd have to pay me quite a bit more to go back to on-prem. You'd have to pay me quite a bit more to go somewhere not using one of the 3 major clouds with all their vendor specific technologies.<p>The speed is invaluable to a business. It's better to have elevated spend while trying to find good product market fit. I didn't understand this until I worked somewhere with a product the market wanted. > 50% growth for half a decade wouldn't have been possible on-prem. > 25% growth for a full decade wouldn't have been possible on-prem.<p>I've been with my current company from $80MM ARR to $550MM ARR. We've never breached more than 1.5% of income on total cloud spend. We've been told that's the lowest they've seen by everyone from AWS TAMs to VC/PE people. It's because we're cloud native, we've always been cloud native, and we're always going to be cloud native.<p>You've gotta get over "vendor lock-in". With our agility it's not really a thing. We could move to another cloud or on-prem if we really wanted. Wouldn't be a huge problem moving things service by service over... though we've had a few more recent changes that would be troublesome, we'd be able to work around them because we're architected well.
I’d say $500k per year on AWS is kind of within a dead man’s zone where if you’re not expecting that spend to grow significantly and your infra is relatively simple, migrating off may actually make sense.<p>On the other hand maintaining $100K a year of spend on AWS is unlikely to be worth the effort of optimizing and maintaining $1M+ on AWS probably means the usage patterns are such that the cloud is cheaper and easier to maintain.
How are such savings not obvious after putting the amounts in an Excel sheet, and spending an hour over it (and most importantly doing this <i>before</i> spending half a million/year on AWS)?
In my opinion the story here is that AWS allowed them to quickly build and prove out a business idea. This in turn afforded them the luxury to make this kind of switch. Cloud did it's job.
These real world stories are always fun to hear because they trigger such defensiveness from for those who are invested in "cloud computing". This so-called cloud computing seems to have a strong need for a cheering section (hype) and a jeering section (sniping anyone who questions its value). Bring on the jeers.
Curious how much was spent on the migration? I skimmed but didn't see that number.<p>> Server Admins: When planning a transition to bare metal, many believe that hiring server administrators is a necessity. While their role is undeniably important, it’s worth noting that a substantial part of hardware maintenance is actually managed by the colocation facility. In the context of AWS, the expenses associated with employing AWS administrators often exceed those of Linux on-premises server administrators. This represents an additional cost-saving benefit when shifting to bare metal. With today’s servers being both efficient and reliable, the need for “management” has significantly decreased.<p>This feels like a "famous last words" moment. Next year there'll be 400k in "emergency Server Admin hire" budget allocated.
I've said this before, unless you are using specific AWS services, I think it is a fools errand to use it.<p>Compute, storage, database, networking. You would be better off using Digital Ocean, Linode Vultr etc. so much cheaper than AWS, lots of bandwidth included rather than the extortionate $0.08 GB egress.<p>Compute is the same story. 2 VCPU, 4GB VPS is ~$24 using a VPS. The equivalent instances (after navigating the obscured pricing and naming scheme), is the c6g.large is double the price at $50.<p>This is the happy middle ground between bare metal and AWS.
Anyone can comment on server lifetime of 5 years? I would think it’s on the order of 8-10 years these days? Cores don't get that much faster you just get more of them, etc
Disclaimer: Former AWS consultant from banking/Wall St.<p>It's been a big "duh" for 20+ years. Large-scale, consistent loads aren't suited to cloud infrastructure. Mostly its shops that don't care about costs, don't know any better, or lack technical capabilities outsource most of their infrastructure.<p>The use-cases for *aaS are:<p>- Early startups<p>- Beginning projects<p>- Prototyping<p>- Peaky loads on-demand or one-of<p>- Evade corporate IT department<p>AWS and VPSes can also be ill-suited for personal use if you live in a major city with bottom tier, cheap datacenters that can rent a 1 GbE uplink, a PDU plug, and 4U. For anything substantial, it's not hard to lease some dark fiber and run (E)BGP.<p><a href="https://www.ripe.net/manage-ips-and-asns/as-numbers/request-an-as-number" rel="nofollow noreferrer">https://www.ripe.net/manage-ips-and-asns/as-numbers/request-...</a>
The math I have always seen is cloud is around 2.5x more expensive than on-prem UNLESS you can completely re-architect your infra to be cloud native.<p>Lift and shift is brutal and doesn't make a lot of sense.
Now get rid of k8s, put the system back together into a monolith, optimize it, get 3 mac minis with gigabit ethernet and save yourself another 65k. It's an uptime monitoring tool...
I would go with managed bare-metal, it's a step up from unmanaged bare metal cost wise but saves you on headaches from memory, storage, network, etc issues.
Very nice !
(Hard time finding the author/contact info though...)<p>At "Storage and LoadBalancers" the NFS link point to <a href="https://microk8s.io/docs/nfs" rel="nofollow noreferrer">https://microk8s.io/docs/nfs</a>
Should be <a href="https://microk8s.io/docs/addon-nfs" rel="nofollow noreferrer">https://microk8s.io/docs/addon-nfs</a>
If you're using AWS primarily as a place to run virtual machines, you probably should be somewhere else. The largest benefits appear when you primarily use cloud-native things: lambdas, sqs, dynamodb, etc. Of course these will constitute a vendor lock-in, but that's usually an acceptable compromise.
I'm in disbelief at how much people pay for aws. At a startup the costs were close to $500 per dev account. Just to keeping a few lambda functions and dynamo tables.<p>Rolling a droplet, load balancer and database costs like $30.
Cheaper price, lower redundancy:<p>>single rack configuration at our co-location partner<p>I've got symetrical gigabit static ipv4 at home...so can murder commercial offerings out there on bang/buck for many things. Right up until you factor in reliability and redundancy.
Can anyone explain or shed more light on how the "remote hands" part the support would work in bare metal server scenario? Would they run commands you give (or) will they follow your runbook (or) something else (or) all of the above?
Are they accounting for all of the labor costs of managing stuff in their stack that they previously deferred to AWS, like OS imaging/patching, backups, and building your own external services instead of whatever AWS had available?
I’m sure if the stack is simple enough it’s non trivial for most senior plus engineers to figure out the infrastructure.<p>I’ve definitely seen a lot of over engineered solutions in the chase of some ideals or promotions.
what is the market distortion that allows AWS margins to remain so high? there are two major competitors (Azure, GCP) and dozens of minor ones.<p>It seems crazy to me that the two options are AWS vs bare metal to save that much money. Why not a moderate solution?
ServeTheHome has also written[0] a bit about the relative costs of AWS vs. colocation. They compare various reserved instance scenarios and include the labor of colocation. TL;DR: it's still far cheaper to colo.<p>[0] <a href="https://www.servethehome.com/falling-from-the-sky-2020-self-hosting-still-pays/" rel="nofollow noreferrer">https://www.servethehome.com/falling-from-the-sky-2020-self-...</a>
Would anyone be interested in an immutable OS with built in configuration management that works the same (as in the same image) in the cloud and on premise (bare metal, PXE, or virtual)? Basically using this image you could almost guarantee everything runs the same way.
Sorry, but if you're mainly using AWS for compute (ie EC2) and scale minimally you're doing it wrong and probably burning a whole lot of money.
It feels like every comment on this article didn’t read past the first paragraph. Every comment I see is talking about how they likely barely made any money on the transition once all costs are factored in, but they explicitly stated a critical business rationale behind the move that remains true regardless of how much money it cost them to transition. Since they needed to function even when AWS is down, it made sense for them to transition even if it cost them more. This may increase the cost of running their service (though probably not) but it could made it more reliable, and therefore a better solution, making them more down the line.
I had an out of touch cofounder a few years back, he had asked me why the coworking space’s hours were the way they were, before interjecting that companies were probably managing their servers up there at the those later hours<p>like, talk about decades removed! no, nobody has their servers in the coworking space anymore sir.<p>nice to see people attempting a holistic solution to hosting though. with containerization redeploying anywhere on anything shouldn't be hard.