I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it.<p>This company saved $800k/year. Perfect time to go in-house with this solution.<p>But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-house.<p>At the right time, you make the right transitions.
They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances).<p>AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce your bill and often that means using AWS-specific services. If you want to stay completely "cloud agnostic" you are going to paying more than buying into a "cloud", in that scenario then you absolutely should be looking at dedicated servers. AWS is great because you can bring existing software and just run it in EC2 (or their container stuff if your software is containerized) but the AWS magic comes from using their managed services (also spinning up/down EC2 instances as needed, but if you are running them 24/7 then consider alternatives or at least pay for reserved).
Dedicated hosting providers.<p>I'm so amazed that somehow people completely forget that for literally decades, web host provided dedicated hosting options at fantastic prices.<p>Yes, loooong time ago - to get your dedicated server might have taken a few hours to provision and the instant server access that AWS brought should not be discredited.<p>But large numbers of web host today allow you to programmatically spin up a dedicated web host instantaneously and at a fraction of the cost.
This is the Trillion Dollar Paradox described by Martin Casado. You’re crazy if you don’t start your business in the cloud, you’re crazy if you stay there.<p>My new startup is focused on helping application owners repatriate their workloads into their own infrastructure.<p>Our goal is to solve the network complexity challenges with a fully open network stack (open source software with all of the hardware options you would expect, and some you wouldn’t). The solution is designed to be turnkey and require very little network experience. It will use standard DevOps tools that you’re already using.<p>We’re announcing it in two weeks and will be posting info here on HN!
I'm glad to see more of these types of articles, but at the same time I'm a bit flabbergasted that this isn't obvious for so many people.<p>These cloud providers are, by definition, charging you more than it would cost you to run it yourself. What you get in return is a guarantee of expertise and an ecosystem.
They saved $800K on their AWS bill, but<p><pre><code> - may spend $250K on servers, replaced after 3 years becomes $83k/yr
- may spend $120-250K on extra staff to maintain the infrastructure
- may spend $15K for a cage in a DC
</code></pre>
They still save $452K/yr overall (actual savings 1st year only $285K). It's still a savings for sure, but always keep TCO in mind.<p>The real fun comes later when you outgrow your cage and there's not enough space left in that DC, or they just have shitty service constantly knocking out your racks, and you have to consider splitting your infra between DCs (a huge rewrite) or moving DCs (a huge literal lift and shift). Have been part of both, it's... definitely a learning experience.
I've said this a hundred times and it seems not loud enough.<p>AWS is not cheap because of your server costs.<p>AWS is cheap because of elasticity, velocity (opportunity cost of next feature), and reduced maintenance hours.<p>"The cloud" was never (afaik) was about getting a cheaper VPS. It was about being able to get them on demand, give them back on demand, and generally not have to maintain anything besides your code (and maybe apply updates to datastores / AMIs)<p>Now, if those premises are not true for your startup/business, then AWS is not the tool for you. I didnt see any analysis of ongoing maintenance costs in the 800k saved, but will it take 1-2 FTE engineers to now be more oncall, more server upgrades, more security patches etc? That's easily 1/2 that savings gone already.<p>Edit: for the most part these attributes apply to GCP, Azure, Heroku etc as well, its not just about AWS
So little real information...<p>So the team is now responsible for backups, hardware ordering,.forecast etc?<p>How big is the team now compared to before?<p>Does it scale?<p>If you price it correctly and keep the free tier small, I would either talked to AWS for better pricing or moved to another cloud Provider.<p>S3 on AWS is a total no-brainer, minio on bare metal might mean much more work and a bigger infra team than business actually wants.<p>I would also love to know what optimizations are already in place. Does cloudflare caching work? Are the results compressed on rest? Is geolocation latency relevant?<p>Why even Cassandra? Are websites not unique? Wouldn't a nginx and a few big servers not work?<p>But who knows? The article doesn't tell me that :-(
There is no way this figure is accurate. The annual spend cited of $1,000,000 is purely hypothetical, as admitted here:<p>"However, all this data and processes need to happen on a server and, of course, we used AWS for it. A few years of growth later, we’re handling over 70,000 pages per minute, storing around 560 million pages, and paying well over $1,000,000 per year.<p><i>Or at least we would be paying that much if we stayed with AWS.</i> Instead, we were able to cut costs by 80% in a little over three months with some out-of-the-box thinking and a clear plan."
There is not quite enough information here to be sure, but this article highlights transmission costs. This particular business model involves throwing around big chunks of data just in case they end up being needed and then handing them back out in response to potentially large numbers of requests. That would make this particular usage pattern fit to exactly what AWS is charging the most for. Also many alternative AWS services that can be used to speed up or simplify services are not really going to help with this case.<p>So an alternative way of interpreting this is more along the lines of: We may have saved up to 80% of server costs by moving from AWS, but you almost certainly won't save that much even if a bunch gets spent on developing operations and tools.
I feel like there's a middle step missing from this article (or I just missed it reading quickly) - did they build their own data center? Where are these new non-AWS servers physically located?
In general, last time I looked at AWS it made sense from 2TB to 30TB a month, and under 400k connections a day. If either range was exceeded, than the service ceased to be the economical choice when compared with CDN providers, and colo/self-managed unlimited-traffic options.<p>For example, if you primarily serve large media or many tiny files to clients that don't support http Multipart Types, than AWS can cost a lot more than the alternatives. However, AWS is generally an economical cloud provider, and a good option for those who outsourced most of their IT infrastructure.<p>The article would be better if it cited where the variable costs arose.
"We used Apache Cassandra nodes on each of the servers that were compatible with AWS S3".
What does this even mean?<p>Regardless, starting a new Cassandra cluster in late 2022?! I bet they can save even more by just going with scylladb
I believe this is the use case Cloudflare is really targeting with R2. They recently connected Cache Reserve to R2 to make this even easier. We wrote up a breakdown for S3 vs R2 and found that R2 would be significantly cheaper when the majority of traffic is cached data, <a href="https://www.vantage.sh/blog/cloudflare-r2-aws-s3-comparison" rel="nofollow">https://www.vantage.sh/blog/cloudflare-r2-aws-s3-comparison</a>
We've just finished moving servers from AWS to <a href="https://hetzner.com" rel="nofollow">https://hetzner.com</a> - and saved 10X with servers of double the capability. A great experience so far.
I find this interesting, if nothing else. My first question is what was the opportunity cost of focusing manpower to setting up on-prem infrastructure that now needs maintained? What on the product roadmap was sacrificed/delayed in exchange for the time on this project? What are the projected future hiring costs to maintain these servers (and applications like Cassandra!) going forward? Nothing is free, and at just 4-5 additional hires they will be giving back a large chunk of that $800k to employees. IDK - maybe that's a fare trade-off to pump up the common man with money instead of the establishment.
I found the headline to be misleading. The article is mostly about the migration process (which is interesting), but very little about the details of the cost savings.<p>What does it cost to run their data center? What are the salaries they are paying for internal IT efforts to administer it? Is it an apples-to-apples comparison, e.g. are they load balancing across multiple datacenters in case of an outage?<p>It sounds like this was a good move for Prerender but it's hard to generalize the cost claims to other situations without details.
Perhaps I'm missing it in the OP -- I don't see any mention of what they actually moved to. CoLo? VPS? On-prem?<p>This seems like a key detail when telling people about your migration off AWS.
I expect to see a great deal more of the "cheap and cheerful" AWS migration stories in the future. With the tanking of the market and (apparent) limits to growth being in the forefront, reducing expenses will become more important.<p>Before, it was easy to justify almost any expense with the "we just need to get 1% of this $100 billion market" and now it is "hunker down and do everything you can to be ramen-profitable, in order to survive and thrive".
When the cost of delivering your product/service is mostly compute or traffic, sure, migrating off of AWS is a must once you reach a certain scale.
But for the other 99% [0], where infrastructure is but a small cost, then think really hard if you're willing to trade the engineering effort for the convenience of managed cloud services.<p>[0]: or 90%, or 80% or who cares, but a majority of software services seem to NOT be compute- or traffic- heavy.
(Note: I have never done any professional work in cloud. I could be completely mistaken. Feel free to correct me if I'm completely off-base.)<p>It's a fascinating article, for sure. I would have been interested to hear what their backup strategy looked like though.<p>One of the big benefits of cloud services, that I am aware of, is the assurance that if natural disaster strikes, you don't lose all of your data. I kind of got the impression that, more than anything else, <i>that</i> is what you are paying for. Data protection and uptime.<p>I suppose big enough bills could lead a company to make the kinds of changes that Prerender did, but when that disaster does strike, and it is time to try and recover from a fire, flood, earthquake, etc. the responsibility and <i>speed</i> of getting your customers back online is reliant completely upon your staff - a staff who might be extremely shaken up, hurt, or pre-occupied in taking care of their own affairs. I'm not saying it's not possible, but there is a kind of cost that comes in the form of responsibility. It's a trade off that I would not fault many people from avoiding.
Hi! I’m Todd, the solopreneur founder of Prerender.io and I created that $1,000,000/year AWS bill. I sold Prerender.io to Saas.group in 2020 and the new team has done an incredible job growing and changing Prerender since I left.<p>$1M per year bill is a lot, but the Prerender back end is extremely write-heavy. It’s constantly loading URLs in Chrome in order to update the cached HTML so that the HTML is ready for sub-second serving to Google and other crawlers.<p>Being a solo founder with a profitable product that was growing organically every month, I really didn’t have the time to personally embark on a big server migration with a bunch of unknown risks (since I had never run any bare metal servers before). So the architecture was set early on and AWS allowed me the flexibility to continue to scale while I focused on the rest of business.<p>Just for a little more context on what was part of that $1M bill, I was running 1,000+ ec2 spot instances running Chrome browsers (phantomjs in the early days). I forget which instance type but I generally tried to scale horizontally with more smaller instance sizes for a few different reasons. Those servers, the rest of the infrastructure around rendering and saving all the HTML, and some data costs ended up being a little more than 50% the bill. Running websites through Chrome at scale is not cheap!<p>I had something like 20 Postgres databases on RDS used for different shards containing URL metadata, like last recache date. It was so write heavy that I had to really shard the databases. For a while I had one single shard and I eventually ran into the postgres transaction ID wraparound failure. That was not fun so I definitely over provisioned RDS shards in the future to prevent that from happening again. I think RDS costs were like 10%.<p>All of the HTML was stored in s3 and the number of GET requests wasn’t too crazy but being so write heavy on PUT requests for recaching HTML, with a decent sized chunk of data, the servers to serve customer requests, and data-our from our public endpoint, that was probably 30%.<p>There were a few other things like SQS for populating recache queues, elasticache, etc.<p>I never bought reserved instances and I figured the new team would go down that route but they blew me away with what they were able to do with bare metal servers. So kudos to the current Prerender team for doing such great work! Maybe that helps provide a little more context for the great comments I’m seeing here.
Throwaway here. I work for a startup which runs compute-bound scientific simulations. We are considering migrating from our single self-managed HPC, to AWS EC2. We anticipate using the largest standard instances, with highly spikey usage: frequently none, at times up to 10 concurrent instances, perhaps more later.<p>AWS seems ideal to me because it would let us easily scale up and down as our usage varies. But some of the anti-AWS sentiment in this article has given me pause. Any reason not to do this? Any alternatives I've missed?<p>Our storage and transfer usage will be negligible; it's all compute.
As a side note, I find this:<p>>Do you have any advice for software engineers who are just starting out?<p>>Don’t be afraid to talk with the customers. Throughout my career, the best software engineers were the ones who worked with the customer to solve their problems. Sometimes you can sack a half year of development time just by learning that you can solve the customer’s issue with a single line of code. I think the best engineers are creating solutions for real world problems.<p>to be very good generic advice.
This is unsurprising.<p>The point of AWS is to be flexible. You’re paying for that. It’s easy to start. It’s easy to stop. It’s easy to change capacity.<p>Running your own servers is none of these things. But it is cheaper at sufficient scale. You can’t ignore the labor cost (particularly engineering) however.<p>Where AWS shines is with highly volatile workloads. With your own servers you have to provision for peak capacity. That’s less the case with AWS.<p>No shade on the author of course. It’s great to read things like this.
Some S3 providers can give you free egress when using a CDN.<p>For example backblaze B2 offers free egress through Cloudflare, Fastly, BunnyCDN.<p><a href="https://help.backblaze.com/hc/en-us/articles/217666928-Using-Backblaze-B2-with-the-Cloudflare-CDN" rel="nofollow">https://help.backblaze.com/hc/en-us/articles/217666928-Using...</a>
Want to add, cloud vendors' proposition is not about the capability; it's a basic requirement and no longer an USP for something to 'work with the cloud'. It's about security. It's about outsourcing the liability so that the manager has less responsibility.
I wish the article went into detail about what hardware they used for each server, what was their disaster mitigation plan, and other considerations that you don't need to worry about with paying for a cloud provider.
OK, so they're now stuck maintaining their own Cassandra cluster. How much does that cost?<p>If it costs you $1,000,000 a year to serve 1166 requests a second, maybe you fucked up.
The company I used to work for. They successfully did cut server costs!<p>...at the expense of 40 eng-years (20 eng over 2 years) spent on the migration.