$45k in AWS charges due to account being hacked to mine cryptocurrency

So, I have a personal AWS account, and setting up cost controls seems like a good idea. Let&#x27;s try it out.<p>&quot;Welcome to AWS Cost Management<p>Since this is your first visit, it will take some time to prepare your cost and usage data. Please check back in 24 hours.&quot;<p>My &quot;cost and usage data&quot; is that I have an S3 account costing ~$10 a month containing my cloud backups of my family&#x27;s high-priority data, for maybe the last 18 months. That&#x27;s it. I just reloaded in case 24 hours was some sort of pessimistic estimate and they actually had it done in a couple of minutes, but the message persists.<p>WTF.<p>I can set up any of dozens of services in thirty seconds and spend thousands, but it&#x27;s going to take 24 hours to understand ~$100 in costs over the course of the last <i>year</i>? Someone could <i>read aloud</i> the entire history of the account, every S3 transaction I did, in that amount of time.<p>I can really feel the love here.

This is exactly the scenario that people post in every thread about AWS failing to implement budgets and hard limits, and every time people always say &quot;but AWS will just forgive the fee!&quot; as an excuse for AWS being so user hostile.<p>So far, this time, they haven&#x27;t. I hope they do for this guy&#x27;s sake, but to not have that as an official policy, or to implement a tech solution, is beyond terrible.

Actually discussed the problems around dynamic billing on HN a while ago, here&#x27;s my comment on the topic, alongside some other platform recommendations, at least as far as VPS hosting is concerned (which is suitable for me since i run a lot of software in containers, but perhaps not to everyone else): <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29339477" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29339477</a><p>From there, a discussion continued, but the problem of what to do when some sort of resource allotment is exceeded, is still not solved.<p>Personally, i think that static allotments (e.g. you get a VPS with X bandwidth per a month, where the port speed will be decreased 10x until end of billing period in case it&#x27;s exceeded) are the way to go for people working on non-critical projects, where downtime or degraded performance is preferable to not being able to buy food or afford rent.<p>Though with some platforms, you still need to protect yourself, either with virtual cards or other means, since there&#x27;s no guarantee that cases like those in this post will be noticed and the fees even waived by the corporations. Of course, the legal aspects of all of that are somewhat unclear. All the more reason to look for solutions with simpler billing, or at least billing alerts in lieu of that.

This happened to me back in 2014, albeit with a smaller bill of around ~$10k.<p>I was pretty inexperienced and I accidentally committed my Amazon credentials to git and went to bed.<p>When I woke up I had a handful of emails from Amazon alerting me to the situation. Back then $10k was almost a third of my yearly income, so it was a pretty traumatic morning say the least hah.<p>When I spoke with an Amazon representative they were very nice about the situation and dropped all the charges, since it was obvious what had happened. I&#x27;m sure that&#x27;s what will happen in this situation, especially given the social media attention.

AWS really needs a hard limit of $50 by default. Any user that needs more needs to explicitly set the new limit only after some type of 2FA preferably an OTP verification. Most of the times these hacks are often due to leaked API keys.<p>These soft limits and email alerts are easy to miss and a lot happens by the time you can stop it.

I remember being very new to all of this (still am) and being absolutely petrified that as I was exploring an Amazon stack that I&#x27;d unintentionally rack up thousands of dollars of charges for an honest mistake. There was part of me that felt so much better about using App Engine at the time; in that case, I had full confidence that the worst I could do is have the app I was building just break if I got to a certain quota limit. That put my mind SO MUCH more at ease. The risk of an accidental cost overrun has kept me from learnings&#x2F;using AWS to certainly a detrimental degree for me, but the fear of something like this happening is very, very real.

I&#x27;ve had two experiences with this. Once due to a credential leak through a public GitHub Gist (PSA: Gists are public). Once through the failure to delete a resource with premium IOPS for a couple months (PSA: It&#x27;s easy to leak resources when developing Terraform scripts on AWS - remember to clean up afterwards.)<p>In both cases, we received a partial credit. Amazon expects you to setup cost explorer and budget controls before doing anything else. These features must be manually activated on every billing account. It&#x27;s kind of maddening actually that even turning on Cost Explorer takes 24 hours.<p>PSA: Turn on Cost Explorer immediately after creating a new AWS account and don&#x27;t give out API keys until you&#x27;ve setup budget limits. If you join a new company in a role where you have access to the AWS billing account, make sure the cost and budget features are enabled.

Doesn&#x27;t AWS let you set a monthly limit? I only see a way to set spending alerts, not a hard &quot;do not spend more than X&quot; limit.<p>There&#x27;s no way on earth I would use AWS on my personal card unless I could set a limit.

To me, the most outrage thing is AWS is charging $45k to mine $800 worth of XMR. Using dedicated servers, it would cost maybe $2,000-$3,000.

I believe this is more by design than anything. AWS makes real money from the big guns where they are spending 100s and 1000s of dollars per month (may be even a Million I am sure). If they put limits, it could hurt their business with these big guns who usually have stricter controls on their end and are not leaking keys or don&#x27;t have 2 factor Auth enabled etc.<p>The challenge is for the smaller guys who are just experimenting with AWS. I see a lot of these stories where they haven&#x27;t even used the account much or at all and then get hacked.<p>Of course AWS can impose limits but they clearly don&#x27;t want to. They would rather refund these cases whenever they feel like.

The large cloud companies seem dead set against creating any type of hard billing limit. Just let me set a magnitude of charges I can ever expect from the relationship, rather than unconstrained open-ended liability.<p>We all know it would be straightforward to implement. Even if the time to shut down systems creates some overage that the provider is unwilling to eat (although I can&#x27;t believe it would cost more than continually refunding these bills), then it would be perfectly fine for &quot;$200 hard limit&quot; to mean start shutting down systems at $100. Just let me limit the damn magnitude!<p>But since they&#x27;ve all decided it&#x27;s good business to be sticking users with bullshit bills, we&#x27;re left to create our own. I&#x27;ve got LLCs on the brain, so I&#x27;m wondering if running one&#x27;s cloud expenditures through an LLC would work.<p>Besides the paperwork overhead, the biggest worry would be piercing of the corporate veil for being undercapitalized. But I hope it would be hard to argue that a company was undercapitalized because it lacked the ability to pay for unexpected expenses that it did not initiate.<p>Does anybody know the legal and practical issues with signing up for these cloud providers through an LLC? Even if you&#x27;re already operating as a company (eg startup), it would still be prudent to sign up through a separate subsidiary. At the very least an extra layer would make for some red tape they&#x27;d have to cut through to press the debt.

How are these companies allowed to provide services without direct customer support. We all understand they don&#x27;t want to pay for the labor, but that can&#x27;t be an excuse.

We are a very large AWS reseller and this drives me bonkers with <i>EVERY</i> Cloud, and I&#x27;ve written about this countless times.<p>There is almost no way what so ever to limit risk and it&#x27;s crazy.<p>Cloud operators state that it is because they don&#x27;t want to shut down popular services or similar - and to be honest, I get this argument - but, it is just awful in practice.<p>Nearly every cloud has cost report delays of anything up to a month for some services and it&#x27;s just a nightmare.<p>I kind of agree that if you have a service costing £5,000 a month - you shouldn&#x27;t limit spend to £5,001 - but, you should be able to limit to £7,500 - enough room to burst&#x2F;grow, but not have risk!<p>All clouds could learn a thing or two from telecoms and rate limits... e.g. if you have services that when not in use have a fixed monthly limit - e.g. storage - then take that out of monthly limit and then use the excess for variable costs.<p>e.g. storage costs £500 a month, compute costs £200 a month... if a client sets a budget of £1,000 - then deduct £700, maybe ~£100 extra for idle transactions (if in a cloud&#x2F;model that charges), then have prepay of £200 for &quot;extras&quot;.<p>I just see &quot;the clouds&quot; do AMAZING new features - all this crazy and cool AI bits, yet they can&#x27;t get basic costs correct - it&#x27;s crazy in my mind.

I just went to set up a hard limit on several accounts. There is email notification. The only actions stop individual services on individual Amazon server locations.<p>There is no global &quot;DROP DEAD&quot; action freezing my account and my liability to Amazon.<p>My ideas for fake notification addresses and their purported actions are likely against HN rules. I&#x27;m just going to leave Amazon AWS.

I feel like whenever this topic comes up, people go back and forth over whether or not AWS should allow you to set a hard limit and what that entails on both sides &#x2F; why it’s impossible.<p>I feel like a better approach would be to specify what services you actually want to run and that requires a separate secret key that you will never use for other development tasks, and you have the ability set hard limits for those services.<p>AWS already has hard limits on things like the number of S3 buckets you can create, that require manually requesting an increase if you want more. Why can’t people say ‘I only want a maximum of 5 EC2 servers, and I don’t want the ability to spin up anything above a t3.large(translated to maybe ~20cents&#x2F;hour per server)’. A similar approach can be taken for other resources.<p>This gets around the issue of ‘what should AWS do if you hit your spending cap’ by allowing you to set an upper bound in the amount of hardware you can actually spin up. It also solves the issue of a compromised dev key or root user account that would likely trivially allow someone to remove a spending limit anyway.

Happened to me, just a bit over $50k. Took a couple hours of work on my part (they asked me to enable CloudTrail and do some additional securing), and they removed the charges.<p>Also, it&#x27;s less of a &quot;hack&quot; than a likely publishing of AWS keys (if you publish them in a public Github repo, they&#x27;re compromised in seconds)

I took his advice and went to the Cost Anomaly Detection console and started to set up an alert for my $0.56&#x2F;mo S3 charges. I want individual alerts, so I have to use SNS. I want a text message for an alert, so I am pushed into creating a 10DLC origin (even while in the SMS sandbox). A 10DLC registration is $50 for a brand or company, and $50 for each &quot;campaign&quot;, plus $15&#x2F;mo for each campaign. Am I missing something?

I haven&#x27;t used my old AWS account in a while, it&#x27;s just sitting dormant. Since it&#x27;s old it&#x27;s linked to my Amazon account (root credentials == Amazon.com credentials).<p>I don&#x27;t want to monitor it and these stories give me some anxiety.<p>I guess I&#x27;m supposed to close my AWS account? I wonder if this is even possible if it&#x27;s the same credentials as your Amazon account.

This happened to me. Luckily it was caught before the bill went over $500, and Amazon refunded all the charges. I have no idea how my account was compromised, and Amazon wasn&#x27;t able to help me investigate. And they made me go through and delete resources across several different regions in a painstaking manual process that took me an entire work day.

They stole is ROOT credentials.<p>This is not about sneaky AWS, this is about a hack. There is not much you can do if someone gets root access to your system.<p>Setting up a billing alert to notify you your aggregate cost exceeds a certain limit takes about 90 seconds. I don&#x27;t understand the overwhelming friction to the AWS billing tools in this thread.

First, the complainant needs to take ownership of poor secrets generation and management. Their account wouldn&#x27;t have been hacked if they used strong password policies.<p>There is also the remote possibility they&#x27;re playing the victim to get out of charges after they realized it wasn&#x27;t economical to mine on AWS.

Annoyingly, AWS won&#x27;t allow you to lower the quotas aka service limits. They can only go up. Even after you completed the project and don&#x27;t need them anymore, they remain forever and create an increased risk.

I&#x27;ve had two machines on Google Cloud that racked up $500 in charges from cryptocurrency thieves. They send me alerts but it is so hard to get through the billing dashboards and understand what is legitimate.

After seeing this I logged into my existing but not really used AWS account and enabled MFA. I did not get any email alert or notification about to change which seems very weird

Can you use a Visa gift card with a $200 amount? Then you could just use a fake name, fake email, done. Need more money? - just add a new gift card.

Just curious, is there any other way to monetize compute&#x2F;memory&#x2F;space power apart from cryptocurrency?

This guy is toast, without evidence to prove he has no relation to this hacker, the charges will not reverse.

Worst part is that the cryptominer probably made $1.32...

Honestly AWS is just a casino; people are blowing ridiculous amounts on RNG to try and find nice hash tables that could potentially make money by manipulating cattle.

I get that some alerts should be sent, that AWS should scan for common scams, react faster to support tickets etc. But how does one get enough access to someone&#x27;s AWS account to create a lambda that mines crypto? Something has an access to your credit card and bills you monthly, yet you do not have 2FA? I would understand if this was some dependency chain attack, but this is just a new lambda with an .sh script. Also what AWS income has to do to with his issue?