My Amazon S3 Mistake

There are a few lessons here:<p>1. Use IAM roles. AWS keys should only have access to the specific functionality they need. Best practice is to <i>never</i> use your root credentials for <i>anything</i>; always use IAM users and use unique keys for each use case, so that you can invalidate and replace them easily. Every time documentation or a tutorial asks you to insert AWS keys, your response should be &quot;let me go create a role for that&quot; rather than &quot;let me go look those up&quot;.<p>2. More importantly, if you ever accidentally publish or leak credentials, don&#x27;t try to clean it up by deleting those commits. <i>Invalidate the credentials immediately</i> and re-issue them.<p>3. Always, <i>always</i> `git diff HEAD` before committing. Know what you&#x27;re about to push up. This isn&#x27;t just a security concern - the number of small, stupid things you&#x27;ll catch that you&#x27;d otherwise end up fixing 15 seconds later is substantial. As a bonus, this incentivizes you to keep your commits small and atomic.<p>You might talk to Amazon directly - they&#x27;ve been known to forgive debts like these in these kinds of circumstances.

&gt; I learned a valuable lesson here though. Don’t trust .gitignores and gems like Firago for keeping your data safe. Open source is awesome, but if you are dealing with anything that can be scaled up to thousands of dollars per hour – at least store it in a private repo if not on your local machine.<p>You learned the wrong thing :-P The lesson here is &quot;always check what you&#x27;re committing&quot;

Hell, even without making the mistake of publishing keys, I&#x27;ve accidentally run up quite large bills for Amazon services; backups that failed to remove old copies was a big one. Instances that were supposed to have been shut down, but for some reason it didn&#x27;t happen (I don&#x27;t know if this is my mistake or a bug at Amazon...probably my mistake).<p>I&#x27;ve simply stopped using Amazon for anything tinkery, because the costs of making mistakes can be tremendous. At least when I make mistakes on my own colocated server, I know it can never cost me more than the $100&#x2F;month I pay to host it. And, storage is practically infinite (4TB hard disks), and I can spin up more VMs than I would ever need for tinkering in 32GB of RAM on our &quot;spare&quot; server.<p>And, when I have needed to use VM with some cloud host...Linode and Digital Ocean and similar may have dramatically smaller toolsets for managing virtual resources than Amazon (probably unworkably so for large deployments), but my mind has a much easier time predicting costs than with Amazon. After being surprised on more than one occasion with a ~$300 bill from Amazon (for running nothing but personal pet projects with no economic return), I turned everything off.

PSA<p>If you&#x27;re running AWS, I highly recommend for all developers and sysadmin to attend AWS free training[0][1]. While you&#x27;re at it might as well get yourself certified[2].<p>You might have senior expertise with system operation and application deployment but sometimes, AWS approach things differently. The essence of the training is to always implement best practices, not just solving problem.<p>Also use the opportunity of AWS event to network with their Solution Architect. Trust me on this one. This worth more than AWS Enterprise Support.<p>[0] <a href="http://aws.amazon.com/training/course-descriptions/architect/" rel="nofollow">http:&#x2F;&#x2F;aws.amazon.com&#x2F;training&#x2F;course-descriptions&#x2F;architect...</a><p>[1] <a href="http://aws.amazon.com/training/course-descriptions/architecting-advanced/" rel="nofollow">http:&#x2F;&#x2F;aws.amazon.com&#x2F;training&#x2F;course-descriptions&#x2F;architect...</a><p>[2] <a href="http://aws.amazon.com/certification/" rel="nofollow">http:&#x2F;&#x2F;aws.amazon.com&#x2F;certification&#x2F;</a>

You can use AWS IAM (<a href="https://aws.amazon.com/iam/" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;iam&#x2F;</a>) to help prevent something like this. AFAIK you can create a sub-account that only has access to specific resources, such as S3, and use the keys from that sub-account.<p>I haven&#x27;t used it much, but it looks like you can be very specific in what you allow, such as only allowing access to a single bucket with S3, or a single domain with SES.

Yeah, I did that, though in my case it was a bitbucket repo I thought was private, but somehow ended up public (obviously stupid to ever be checking in the keys at all).<p>All I had setup was a micro S3 instance I&#x27;d been using for some toy craigslist scraping &amp; hadn&#x27;t touched for months.<p>Then out of the blue I get an &quot;urgent please check your account&quot; email from amazon. Go check the AWS console, and what do you know -- maximum number of maxed-out instances instances churning away with 100% CPU usage in every region on earth. The charges were already about to $50,000 when I turned everything off.<p>I wrote a very, very apologetic email to amazon, and they forgave all the charges, for which I was very grateful.<p>Definitely a learning experience.

And please for the love of all that is secure, use IAM roles. Even for your personal things. it&#x27;s not that hard and you can stop things like this from happening even with the auth credentials.

If the criminals can create a bot to scan for AWS keys, I wonder if github can&#x27;t create a plugin to detect the same and warn the committer or maybe limit access to this data to original committer only. It won&#x27;t be 100% but I bet the bots aren&#x27;t 100% either, so if it covers most of the cases it would still be useful.<p>Or maybe just have a script on local github pre-commit hook?

Lesson 1: Don&#x27;t publish your passwords.<p>Lesson 2: When using AWS then use Billing Alarms[1].<p>It takes about 1 minute to setup and enables e-mail or SMS notifications at dollar-thresholds of your choice.<p>[1] <a href="http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/create-billing-alarm.html" rel="nofollow">http:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;awsaccountbilling&#x2F;latest&#x2F;aboutv2&#x2F;...</a>

Also, if you ever accidentally leak your keys... CHANGE THEM!

The same thing happened to my brother. He&#x27;s been teaching himself software development for a while, and learned the same painful lesson about securing your keys. This was despite the things he did right, like:<p>1. Used IAM roles (but he didn&#x27;t lock down it&#x27;s permissions nearly enough clearly)<p>2. Used two factor authentication.<p><a href="http://snaketrials.com/2014/11/11/espionage/" rel="nofollow">http:&#x2F;&#x2F;snaketrials.com&#x2F;2014&#x2F;11&#x2F;11&#x2F;espionage&#x2F;</a><p><a href="http://snaketrials.com/2014/11/12/espionage-update/" rel="nofollow">http:&#x2F;&#x2F;snaketrials.com&#x2F;2014&#x2F;11&#x2F;12&#x2F;espionage-update&#x2F;</a><p>I should note Amazon forgave the money his account owed, given it was his first time making this mistake on their systems. Amazon told him the servers were probably being used to mine bitcoins.<p>edit - Oh, and on his blog he says it was $2000 over 12 hours. It turned out to be $5000 over 2 hours!

This seems to be bad design on Amazon&#x27;s part as well. There should be some kind of manual thresholding in place to make sure this doesn&#x27;t happen.

I must be very old or very paranoid, but I wouldn&#x27;t for one second think of developing from scratch using only public infrastructure.<p>If I want to learn rail, I&#x27;d start by installing it and running it on my local PC. What&#x27;s the point of relying on heroku, aws, etc for just everything? What&#x27;s the point of using github as the main hub for everything? I&#x27;m synchronizing stuff to github explicitly and carefully, my casual, day-to-day operations are all running on my own hardware, network and services. Even my most simple websites run first at home, then are sync&#x27;ed to the web after I&#x27;ve checked everything works fine.

Some old man Java rambling for you. :) (I am ex-Rails actually)<p>one of the big things about Java that at first I thought was tedious but now have come to love is the build process. By creating the &quot;built&quot; project in a new folder, it is like you are guaranteeing that only stuff you manually specify to be in the production code actually makes it in.<p>i know .gitignore is supposed to handle this but it just seems dangerous to rely on source-control related filters when there is really sensitive security info involved.<p>in node i&#x27;ve used grunt &amp; really like that model. The idea being that you have the project run a build then check that into a &quot;build&quot; branch in git &amp; deploy from there. it&#x27;s also nice because i can uglify etc. if i&#x27;m not worried about preserving the source.<p>I guess for distributed coding this is tough because you want your peers to have access to the raw source + build instructions, but if I were working on a really security-sensitive project (I do enterprise so I&#x27;m starting to understand the need for extra precautions) I would probably only distribute code that goes through a (perhaps even thin&#x2F;transparent) build process &amp; then figure out a way to integrate changes back into the source branch.<p>I know it must sound like overkill to the open source world &amp; especially for Rails projects, where there ordinarily is no build stage, but it seems like there is some level of danger in distributing a source control repo where you are relying on you + any collaborators to properly configure .gitignore and never accidentally check anything sensitive into the repo. I am decent with git but no expert, i still sometimes accidentally stage things I shouldn&#x27;t when initially checking in a .gitignore -- and i have much more faith in myself than some of my peers, hah.

I think a lot of the comments here are good - but a thing to keep in mind with github is that their free repo hosting is by default - public. Which means that your code is indexed and searchable - by anyone visiting github including anonymous users. They do give you free private repos if you are a college student or professor.<p>I am a big advocate of self hosting - not because services like github suck (in fact I&#x27;m a big fan of github, bitbucket and other services) but you have more control over your own code (that and you can setup private repos).<p>Here is a small list of self hosted solutions (I forked indefero into srchub - many don&#x27;t like the google code feel but I personally like it for the simplicity and the fact I can easily fix&#x2F;modify&#x2F;add on to it): <a href="http://softwarerecs.stackexchange.com/questions/3506/self-hosted-replacements-for-mercurial" rel="nofollow">http:&#x2F;&#x2F;softwarerecs.stackexchange.com&#x2F;questions&#x2F;3506&#x2F;self-ho...</a>

There really ought to be a way to <i>cap</i> your bill. Alarms aren&#x27;t enough when large bills can be run up overnight.

I documented a similar experience just over a year ago[1].<p>I ended up helping give a talk about my experience at the Amazon Summit in Sydney. I hope I made a good cautionary tale to the devs&#x2F;ops&#x2F;managers attending.<p>[1]<a href="https://news.ycombinator.com/item?id=6911908" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=6911908</a>

In addition to the steps mentioned above, one thing that has kept me safe from my own heedlessness is to never, ever store credentials in a source code tree.<p>If your project is reading credentials from a file, rewrite it so it reads them from environment variables.<p>Most IDEs make it very easy to do that, and Python&#x27;s virtual environments can do that work for you. Yes, it takes more effort, and sometimes it will be a little convoluted[1].<p>However, it&#x27;s well worth the effort as you will have a system that you can put your faith in rather than having to double check every time in order to make sure you&#x27;re not about to inadvertently commit your API keys.<p>[1] Example of my own: Pycharm does not read variables stored in a virtual environment&#x27;s configuration, so I have to set them twice.

&gt; Unfortunately the Rails tutorial is pretty bland so about half way through I decided to snoop around to see what my options where. Somehow, almost by chance I ended up a subscriber on Lynda. Lynda only has one Rails tutorial, but its pretty in depth and is backed by a five hour Ruby tutorial.<p>I had the situation in reverse. I found the Lynda.com rails tutorials first (in 2011) and I found it lacking compared to Rails Tutorial. I&#x27;m not sure what&#x27;s the situation now, from Lynda.com website it seems the author (Kevin Skoglund) has updated the tutorials for Rails 4.

I thought new AWS accounts came by default with a 20 instance limit...

&gt; Actually hooking into the API was very straightforwards, and didn’t take more than an hour to set up.<p>I have learned that if this is the case, I&#x27;m doing something wrong with my AWS setup.

I used some app to upload a 250GB archive to Amazon Glacier. Turns out the app was having difficulties uploading it all at once, and its queue functionality sucked. I explained this to Amazon, since they were suggesting that I use that app to begin with. Turns out I ended up using terabytes of data usage just getting everything up to Glacier because the app was faulty. They tried to bill me something like $1500, but I never paid for it and just had my data removed immediately.

Maybe Github can reject commits by default that contain phrases of magic keys&#x2F;known strings and file combinations that contain passphrases.

Is it not possible to setup some sort of dollar value or bandwidth maximum and freeze the account upon reaching that value?

mildly sensational conclusion. his real lesson should be &quot;this is how i learned how to use github.&quot;

Is there an easy way of putting a strict limit on the amount of money I&#x27;m willing to spend on AWS?

The expensive mistake was to use Amazon at all. An own domain name costs about $1&#x2F;month, an OVH root server less then $10. Install a minimal Debian and Linux Containers on it, and expand your own cloud, if you need it, e.g. by extending with cheap Hetzner servers.

Amazon has been really proactive in protecting against these kinds of things. They seem to be searching the web constantly for API keys, because they&#x27;ll send you emails that say &quot;hey we found your key here, you better do something about that&quot;.

The same thing happened to me. Almost exactly. Rails again. S3 bucket for images. Following along with 1monthrails this time. Pushed the key, fell asleep, awoke to Amazon warnings and a $2000+ bill. Also removed.<p>I wonder how often this happens. Are they mining bitcoins?

There seems to be some serious confusion here if this dev thinks you can spin up EC2 instances with the S3 API.<p>Perhaps one lesson here aside from keeping keys outside of public repositories is to learn how an API works (IAM, arns, etc) before using it.

I&#x27;ve said this before, but AWS billing support is usually quite sympathetic to your situation if you&#x27;ve made a mistake. They&#x27;ve dropped $120 in spot instance charges I had which went way over what I expected.

&quot;Turns out through the S3 API you can actually spin up EC2 instances&quot;<p>Can you really use the S3 API to spin up EC2 instances? or is the guy just mixing the fact that the credentials he used for S3 can be used for other AWS APIs ?

I&#x27;m not at all surprised to learn of bots cruising github looking for keys. I think a good lesson is that if you ever accidentally expose your API key - revoke or delete them immediately and generate new key.

I&#x27;m surprised no one else mentioned the Heroku mistake of putting config in a file instead of an environment variables. Settings like API keys should always be in env vars on Heroku, not in a config file.

Damn! I know one thing for sure after reading this, I am not even trying AWS until I really really know what I am doing. For now I&#x27;ll stick with VirtualBox and DigitalOcean.

Had a very similar thing happen to a co-worker... accidentally pushed an AWS key to github, and over $7,500 in charges before it was noticed.<p>Fortunately it was forgiven.

This happens very often. Like many others have recommended, disable the global AWS keys and use roles.

The github help page* literally states &quot;Danger.. If you committed a key, generate a new one.&quot; In a box. In Red.<p>I&#x27;m not sure how much clearer GitHub could make that.<p>*<a href="https://help.github.com/articles/remove-sensitive-data/" rel="nofollow">https:&#x2F;&#x2F;help.github.com&#x2F;articles&#x2F;remove-sensitive-data&#x2F;</a>

I wonder how much crypto they mined? Probably $1 worth but still worth it for them!

Always put your API keys in an environment variable or ~&#x2F;.

mmmh. Using s3 key&#x2F;secret to launch ec2 instances ? Does anybody have the pointer to the docs for that ?

&gt;<i>Over the holidays, I opted to try to teach myself Ruby &amp; it’s companion Rails.</i><p>shouldn&#x27;t it be &quot;its&quot;?

Why do people use this? for $3000 you can run like 6 4ghz machines with 12 gb ram out of your house with like 10mbs up and 50mbs down. I seriously believe the % of customers of amazon computing that actually need it is like 1%.

why doesn&#x27;t Amazon just ask you on sign-up if you&#x27;re going to be mining coins, and if you say no, require specific authorization from you (outside of the usual key) to start doing so? Then until you authorize it, they can just not run mining instances on your behalf. surely it&#x27;s easy for them to tell when this is being done?<p>EDIT: this got downvoted, but I stand by it, plus it&#x27;s a question, so you could reply and answer it. In my thinking it&#x27;s the same reason there&#x27;s a daily ATM withdrawal limit <i>set by default</i>. You can lift it, but it&#x27;s there to reduce incentive (payoff from trying to see your PIN and then stealing your ATM card.) the current policy is like the bank calling you and saying, &quot;ummm, I hope you know you&#x27;ve already withdrawn $7,000 and seem to be continuing.&quot; Given that bitcoin is (literally) cash, it seems to me saner to not run bitcoin mining instances by default, unless you authorize it specifically. or can they not tell?