Usually our monthly fee won't exceed 1,000 dollars. We discovered last month's bill is almost 3,000, and for this month up till now it's already over 200,000.<p>We collected the evidences and filed police report. The bill is paid through a distributor, anything we ask about the reduction of payment, the distributor just passes it on to Microsoft. I feel if we don't find a way to talk to Microsoft, we will just end up paying the whole thing.<p>Many of you might think we screwed up, we pay up, but I think it's more like a stolen credit card situation, we can negotiate with the bank. How do I go about this?
Step 1: Read your cloud services contract with Microsoft very carefully. What does it say about your liability for fraud?<p>Step 2: Read your business insurance policy very carefully. What does it say about fraud coverage? What are the limits and exclusions?<p>Step 3: Unless 1 or 2 makes it real clear the business is not liable, get a lawyer.
That really sucks and is the risk of using cloud solutions with no spending limit and a lack of monitoring.<p>You should still have someone to keep an eye on it when using cloud solutions. And when you already have someone to keep an eye on it there's a good chance you might be better off managing the infrastructure yourself.
>but I think it's more like a stolen credit card situation<p>How did the account get compromised? What was the nature of the attack (e.g. cryptocurrency mining, expensive egress traffic for file hosting, etc.)?<p>Every (consumer) credit card I've seen requires you to take reasonable steps to keep the cards secure to be eligible for fraud protection (e.g. changing the PIN if compromised, not lending it to people, alerting the issuer ASAP in case of suspected fraud, etc.). I do not use Azure but I would imagine that it works the same way - that is, if you fail to follow basic security precautions (enabling MFA, not using shared accounts or passwords that have been known to be compromised in a leak, etc.) you'll probably end up stuck with the bill. Hopefully you had things reasonably well secured.
This is a conundrum. On one hand, I understand how frustrating something like this can be. But on the other hand, your cloud provider did provide those services that you're being billed for. So they did incur costs, why would they just eat those costs?<p>Unless they're somehow at fault by exposing your credentials or making it easier for hackers to log in without 2FA or something of that nature.<p>If you're using a credit card to pay (though can't see a credit card having a 200k limit, even business) you might want to see if they can help (though it's not the credit card itself that was stolen, so it's unlikely they'd cover you). Otherwise, I'd imagine you're SOL unless you have some other insurance you can rely on.
I’m sorry to hear this, this is a tough situation.<p>Microsoft might, but are unlikely, to help you out.<p>Similar situation with your bank. Neither face a legal obligation to help you, just potential bad PR if they don’t.<p>Your best bet may be bankruptcy. It sounds terrible, but assuming you have an LLC/Ltd company, you can clear out your coffers, wind up, pay them pennies on the dollar, if anything, and start a new business. You may need to go through an lawyer or administrator depending on bankruptcy laws where you are.<p>I’ve taken a client through this, after a similar situation - they ended up with a vast bill to a supplier brought about by someone else using their credentials, and the supplier not being willing to budge. It cost about a week of time and about $2k in legal fees.<p>I’ve also been on the receiving end, where I presented a legitimate invoice and rather than pay the client reincorporated and kept the IP - which sucks, but Microsoft will be insured against insolvencies, so I wouldn’t feel bad about it. You’re just allowing their insurer to help everyone out.
Please setup billing alerts, know what your daily spend should be, add a little for if things grow a little unexpectedly. But you should absolutely be getting alerts if your spend is out of the ordinary for > 2 hours.
Don't pay it. Send them notice, by registered letter, that the charges are fradulent. If a credit card was charged, try to initiate a chargeback/fraud claim.<p>Once you pay it, you lose all leverage. You're much less likely to ever get any money back.<p>Probably consult with a lawyer.<p>Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In my experience, companies are pretty willing to forgive fraudulent charges if you don't have an unusual history of them.
Try to contact Microsoft support immediately.<p>Don't rely on the distributor/vendor, they act very slowly.<p>You're a customer of Azure, you can contact them by any mean, the fact you pay through a distributor doesn't change that relationship.<p>So I would open a Azure support, and also will try to find Azure team on Twitter/Hacker News etc and contact them politely for help.<p>There is no way you would have to pay this bill. They will sort out something or even waived it if it's the first time.
That's unfortunate situation. It happened to me once before (though, we was using AWS that time. And, I believe the cost was smaller than the one you have right now).<p>What we did to recover the cost was to contact the account manager for our region at the time. So, maybe you could have better luck trying to find the particular person in linkedin. Or, have you tried opened a ticket from Azure console?<p>Nonetheless, I hope after everything has been settled down, you won't fire anyone (and treat it as learning opportunity)
I'm quite surprised that there isn't some kind of monthly budget control. For every new project I set the budget to be 4-5x my expected expense.
I asked microsoft support to vaive the past two months of billing because I left open a database cluster which I created for testing purposes. They promptly replied, took me through the steps and vaived the bills.<p>So maybe just file a support ticket, or have your distributor file a ticket for you?
When that happened to us, we found an article showing Tesla got hacked the same week as us (was aws) and they got the money back, so why not us?<p>We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone.
I can't help you with the legal side of things, but moving forward I advise hiring some security-aware infra guy. The root cause of most of these incidents is some human being incompetent (leading to things like poor security and relying on manual processes) or reckless.