How We Saved $132k a Year With an IT Infrastructure Audit

I will add another suggestion, if you use S3 at all... one of your largest costs is likely the bandwidth. Have you considered just placing caches in front?<p>I took a $100 per month S3 bill down to $5 per month by simply having existing Nginx servers enable a file cache.<p>It does help that I never need to cache purge (versions are saved in S3 and become URLs), but it was super trivial to just wipe out $95 per month of cost for zero extra spend.<p>My current setup is:<p>S3 contains user photos, web app (not on AWS) handles POST&#x2F;GET for S3 (and storing local knowledge), Nginx at my edge has a cache that is currently around 28GB of files, and then CloudFlare in front of all of this saves me around 2TB of bandwidth per month.<p>The real gotcha for me is that I was relying on CDNs for my cache, but when CDNs reach 50+ PoPs I was starting to see multiple requests for the same thing as a result of people in different cities requesting a file. So the Nginx cache I&#x27;ve added mostly deals with this scenario and prevents additional S3 cost being incurred.

A few people in the comments are saying &#x27;isn&#x27;t better to just setup your own SQL server instead of RDS?&#x27; and similar. I don&#x27;t want to post a reply to each, so I will say it here.<p>While I can totally sympathize from a programmer point of view (setting up, tweaking stuff and all is a great fun), but you need to ask yourself whether it is in the interests of business to do so. Especially if you&#x27;re working in a small team with no dedicated infrastructure staff or a startup with a short runway and a lot of urgent user facing changes.<p>Doing something on your own (e.g. setting up your own alternative to S3, or configuring your own SQL servers), comes with a cost and it&#x27;s not only the programming&#x2F;initial setup time. It&#x27;s also opportunity cost (instead of setting up a server, I could, for example, analyze some user data); maintenance (more things to worry about which you can outsource); skills set required to run the infrastructure (running your own SQL cluster requires more knowledge, more training than running one on RDS), etc.<p>So is it in the interest of the business to run your own infrastructure?<p>If you have thousands of servers and spending millions on it - probably, but then probably you can make an attractive deal with GCE or AWS :)<p>If your application needs some complex performance related stuff which is harder to do in the cloud (e.g. some custom hardware or whatever), then again, running your own infrastructure might be better.<p>But if you are like the majority of the companies&#x2F;products (you just need infrastructure to run reliably and performance should be just good enough), using AWS and friends might make a big difference.

If saving money and giving your users a better experience are priorities then, <i>in general</i>, moving off AWS is worth considering. When it comes to EC2, spot instances or maybe you&#x27;re doing it wrong.<p>Conservatively speaking, without bandwidth, you&#x27;re looking EC2 costing 2-4x more than dedicated while being 2-4x slower. This does depend on specific workload, and the gap <i>has</i> been closing (conversely, I&#x27;ve seen specific workloads be worse than 4x slower).<p>I know RDS is convenient. But learning how to setup and manage your own database is actually a fundamental skill that will serve you well. All learning can be seen as an opportunity cost, but this is one that will let you save money every month, and give your users a better experience.

One thing I found surprising on reading this is that they essentially spent $10k and two months ($5k listed as saved * 2 months) to figure out they weren&#x27;t using their logging infrastructure any more. Wish I could be that gung-ho with resources!

The title should really be &quot;How we stopped wasting $132k a Year With an IT Infrastructure Audit.&quot; The article certainly leaves the impression that a lack of processes, procedures and basic change management left you in a position where you were wasting 132k a year. As I read it the article shows some recognition of that fact but a false dichotomy between innovation, time to market and change control is used to justify not properly addressing the issue. Thus it will most likely strike again but in a more painful manner.

I wonder where the tipping is at which it&#x27;s more economical to own&#x2F;lease your infrastructure instead of AWS&#x2F;Azure&#x2F;etc.?<p>The company is certainly large enough to have their own infrastructure team.<p>But granted, migrating an entire system that makes extensive use of the AWS ecosystem is anything but trivial.

The money saved isn&#x27;t nearly as important as the reduced complexity and exposure gained from a regular spring cleaning.

Am I the only one who initially read this as &quot;How We Saved $132k a Year With an IT Infrastructure Adult&quot;? Having an IT Infrastructure Adult is essential, after all.

&gt;For the longest while we’ve used fluent.d[sic] to log events in our systems.<p>As a maintainer, glad to see Fluentd there =) If folks have questions on Fluentd, I&#x27;d be happy to answer them here.

Will add my personal experience regarding RDS .<p>If you are using RDS with provisioned IOPS , you can reduce your bill dramatically by downgrading to General purpose SSD, I know certain applications might really need the dedicated IOPS, but its better if you monitor your read &#x2F; write rates and decide accordingly.

So they put a team of engineers on a problem and managed to save an amount roughly equivalent to the salary of one of those engineers? Also, don&#x27;t AWS resources generally get cheaper with time?

The amount saved affords an extra C-level hire. What intrigues me though, is what required them jumping from 25 to 80 people.

One thing that really stands out is they aren&#x27;t running reserved instances. That must be incredibly expensive.

Any way this could be done with Microsoft Azure as well?

I was able to save a small, niche web hosting company with it&#x27;s own proprietary CRM around $112k annually. I also greatly improved security in their shared hosting setup and automated code deployments and new customer onboarding.<p>I was able to do this by using Edgecast CDN as a caching proxy for all anonymous traffic. This reduced the load on their servers greatly and we were able to decimate the number of servers required. Rackspace servers are incredibly expensive and this represented a big savings.<p>We could have cached the pages in other ways, but this had the added benefit of serving anonymous requests from edge nodes and this reduced page load time by a great bit.<p>It was a big migration with sometimes maddening constraints imposed by business necessities and technical debt, but in the end we were able to eliminate a good bit of that debt.<p>The most frustrating part of the process was having to deal with sales reps that kept trying to push &quot;cloud&quot; solutions as panacea to all scaling challenges.<p>The bandwidth and storage costs from using something like S3 would have been atrocious. The rackspace &quot;cloud&quot; solutions all would have had unacceptable latency problems.<p>And it would have required impossible code rewrites. One of the requirements for the project that was incredibly frustrating was that we could not force updates of the PHP CRM to any particular client. We offered an upgrade path, but we had dozens of different versions of the software running on the servers, along with Wordpress and other PHP&#x2F;MySQL apps installed at customer request.<p>Shared PHP web hosting is one of the most difficult environments to work in. Each account was a petri dish of whatever customers uploaded via their docroot FTP access.<p>I pushed through a lot of changes to eliminate that practice and lock users down to FTP access for directories that would not execute PHP.<p>I also had the company move all Wordpress installs to Flywheel, to eliminate all the maintenance and security implications for Wordpress to a company that focused on just that. This allowed the company to focus on it&#x27;s own CRM and nothing else.<p>All of it came at a really key time because competitive pressure from squarespace forced the company to drastically reduce prices.<p>When I pitched the original idea for the project, the entire internal team didn&#x27;t tell me about the security issues, the multitude of versions running, the full FTP docroot access or even the existence of Wordpress on the servers.<p>When I discovered how FUBAR the entire setup was, as a contractor it would have been easy for me to bail, and probably personally healthier for my stress levels (I took off three months after it was all over to relax), but I stuck with them and brought them through the entire process to a successful conclusion.<p>I&#x27;m pretty proud of that project. I pulled it all off under some of the most difficult and irrational conditions one could imagine.<p>And the ROI for the company was insanely great.