Ask HN: One of our Azure accounts was hacked – how to negotiate the bill?

[frogcoder]

Usually our monthly fee won't exceed 1,000 dollars. We discovered last month's bill is almost 3,000, and for this month up till now it's already over 200,000.

We collected the evidences and filed police report. The bill is paid through a distributor, anything we ask about the reduction of payment, the distributor just passes it on to Microsoft. I feel if we don't find a way to talk to Microsoft, we will just end up paying the whole thing.

Many of you might think we screwed up, we pay up, but I think it's more like a stolen credit card situation, we can negotiate with the bank. How do I go about this?

[nwiswell]

Step 1: Read your cloud services contract with Microsoft very carefully. What does it say about your liability for fraud?

Step 2: Read your business insurance policy very carefully. What does it say about fraud coverage? What are the limits and exclusions?

Step 3: Unless 1 or 2 makes it real clear the business is not liable, get a lawyer.

[akerl_]

Business insurance? That doesn’t sound like part of a minimum viable product.

[dhx]

Relevant agreements are perhaps one of:

1. OP <--> Partner (confidential agreement) <--> Microsoft (MPA)[1]

2. OP <--> Microsoft (MCA)[2]

3. OP <--> Microsoft (MOSA)[3]

The different types of agreements have different limitations of liability clauses. What OP wrote indicates a "partner" is involved and if this is the case, Microsoft have essentially shifted liability for fraud and billing non-payments onto the "partner"[4], who would then either wear the cost or try to shift this liability to the OP. It's not that straightforward though as any of the three parties could have a share of liability, and the "partner" would be very unlikely to want to get in a dispute with Microsoft as this would impact their other business. Liabilities are possibly also impacted by default spending limits and caps that are imposed by Microsoft on different services[5].

Allowing a $200,000 bill for one month (a 200x increase) has the appearance of being very poor financial management from the "partner" as they're potentially going to be stuck with unsecured $200,000+ liabilities from their customers if the customers became insolvent. I suppose it is possible the OP and "partner" have a bank guarantee in place to cover at least $200,000 but I'd hazard a guess they may just try to rely on an insurance policy instead to cover these rare events.

[1] Microsoft Partner Agreement (MPA): https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE...

[2] Microsoft Customer Agreement (MCA): https://www.microsoft.com/licensing/docs/customeragreement

[3] Microsoft Online Subscription Agreement (MOSA): https://azure.microsoft.com/en-au/support/legal/subscription...

[4] https://learn.microsoft.com/en-us/partner-center/non-payment...

[5] https://azure.microsoft.com/en-us/support/legal/offer-detail...

[matkoniecz]

If not done already, prepare off-cloud backup and consider migration plans.

There is some risk that they will terminate your account.

[NorwegianDude]

That really sucks and is the risk of using cloud solutions with no spending limit and a lack of monitoring.

You should still have someone to keep an eye on it when using cloud solutions. And when you already have someone to keep an eye on it there's a good chance you might be better off managing the infrastructure yourself.

[highwaylights]

Not really sure how this exact scenario works but if their account was legitimately hacked couldnt the hacker just remove the caps?

[ownagefool]

So it depends how and what gets hacked.

The type of company posting about getting hacked like this is probably using the root / admin accounts to do most things. Their lowest hanging fruit and biggest wins would probably MFA, then SSO.

However, IAM is generally powerful enough to allow you to configure what an account can do.

So best practice, you also want to think about how you're going to lose credentials.

- Sharing passwords across services - leak of your .dotfiles, either by having your laptop pwned, or uploading your .dotfiles to a public repo as a backup or something. - Accidently pasting into the wrong window or something.

SSO & MFA defeats all of these with exception that your sts token will be signed for 1h in those .dotfiles when you auth yourself. I'm not sure what happens if you remove it from the token from the device, but the device itself being compromised would allow someone to piggy back your session.

Ergo, you move to least privilege access, and then if your laptop, server, or ci/cd runner gets hijacked, then it's only able to do whatever it was allowed to do in the first place.

The last part is you need to detect the misuse.

When you have least privileged access, and a pretty locked down account, the hope is when a session is hijacked, the attacker will attempt to use the credentials and get an access denied. This should allow you to detect and remediate the reason for leak.

Obviously this turns your cloud install into a lot more work, and you still also need to look at maintaining and patching the actual services so they're not compromised in the first place.

[Cthulhu_]

In theory, yes. Which brings us to the question of liability in the case of hacking.

On the one side, if you got hacked to that degree - root account, MFA, e-mail, etc - you really fucked up, to be very blunt.

On the other, it's down to Microsoft to provide good security and protections - e.g. spending limits, with a "contact us" and mandatory waiting period if you're about to go e.g. 10x over what you normally pay.

Banks (at least here) offer you a kind of insurance; if you get hacked, they can lock your account and return you your money. Their reasoning there is that they messed up and didn't make it obivious enough that you're about to, for example, send X amount of money away. (this is one reason why cryptocurrencies will never replace banks)

[imadethis]

Removing the caps should trigger an out-of-band notification such as an email to all stakeholders, ensuring the legit admins will be notified. This should also be the case for admin removal or disabling billing alerts. I have no idea if Azure actually does this though.

[waste_monk]

>but I think it's more like a stolen credit card situation

How did the account get compromised? What was the nature of the attack (e.g. cryptocurrency mining, expensive egress traffic for file hosting, etc.)?

Every (consumer) credit card I've seen requires you to take reasonable steps to keep the cards secure to be eligible for fraud protection (e.g. changing the PIN if compromised, not lending it to people, alerting the issuer ASAP in case of suspected fraud, etc.). I do not use Azure but I would imagine that it works the same way - that is, if you fail to follow basic security precautions (enabling MFA, not using shared accounts or passwords that have been known to be compromised in a leak, etc.) you'll probably end up stuck with the bill. Hopefully you had things reasonably well secured.

[gigel82]

This is a conundrum. On one hand, I understand how frustrating something like this can be. But on the other hand, your cloud provider did provide those services that you're being billed for. So they did incur costs, why would they just eat those costs?

Unless they're somehow at fault by exposing your credentials or making it easier for hackers to log in without 2FA or something of that nature.

If you're using a credit card to pay (though can't see a credit card having a 200k limit, even business) you might want to see if they can help (though it's not the credit card itself that was stolen, so it's unlikely they'd cover you). Otherwise, I'd imagine you're SOL unless you have some other insurance you can rely on.

[sam0x17]

> why would they just eat those costs?

Beacuse the public indignation directed at cloud companies who don't always eat the costs in these situations vastly outweighs the cost of simply eating these costs, at least for cloud companies at the top tier of economies of scale (AWS, GCP, Azure, etc)

If AWS didn't always eat costs like this, startups might think twice before using AWS, etc, etc.

[kelnos]

Exactly this. Cloud providers have to acknowledge that building software on their platforms is necessarily complex, and inevitably bugs can cause extremely undesirable behavior. These providers are already charging a premium for instant, on-demand provisioning and nearly limitless pay-as-you-go resources, and know that there are too few guardrails in place to prevent accidental runaway situations.

"Goodwill" has value to a corporation. Taking a hard line against legitimate mistakes that anyone (yes, anyone) can make costs them goodwill, and costs them customers.

And beyond that, while accidental/fraudulent usage doesn't cost them $0, the services are marked up to the point that they probably doesn't really lose that much by forgiving the charges.

[madaxe_again]

I’m sorry to hear this, this is a tough situation.

Microsoft might, but are unlikely, to help you out.

Similar situation with your bank. Neither face a legal obligation to help you, just potential bad PR if they don’t.

Your best bet may be bankruptcy. It sounds terrible, but assuming you have an LLC/Ltd company, you can clear out your coffers, wind up, pay them pennies on the dollar, if anything, and start a new business. You may need to go through an lawyer or administrator depending on bankruptcy laws where you are.

I’ve taken a client through this, after a similar situation - they ended up with a vast bill to a supplier brought about by someone else using their credentials, and the supplier not being willing to budge. It cost about a week of time and about $2k in legal fees.

I’ve also been on the receiving end, where I presented a legitimate invoice and rather than pay the client reincorporated and kept the IP - which sucks, but Microsoft will be insured against insolvencies, so I wouldn’t feel bad about it. You’re just allowing their insurer to help everyone out.

[kelnos]

> Microsoft might, but are unlikely, to help you out.

Is that true? I have no experience with Microsoft, but I've heard quite a few stories of Amazon crediting AWS accounts when customers write in to say their account was compromised. Or even cases when the customer themselves screwed up some permissions in a way that ended up costing an arm and a leg. Hard to believe this practice would be unique to AWS.

[madaxe_again]

You hear the stories with the happy outcomes - I would wager this is selection bias at work, as I’ve seen several instances firsthand where the outcome has been that the bill stands.

It’s still worth trying, as a first resort, of course, but it isn’t something I’d count on.

[sarlalian]

Please setup billing alerts, know what your daily spend should be, add a little for if things grow a little unexpectedly. But you should absolutely be getting alerts if your spend is out of the ordinary for > 2 hours.

[poly_morphis]

Slightly unrelated, but I do this for everything involving my personal finances. All of my credit cards have mobile apps that instantly push notifications whenever they are used. My bank app sends me notifications for any withdrawal or deposit. I even have alerts for my investments. I guess I'm just adamant in knowing exactly where (and when) my money comes or goes.

[kelnos]

I was really happy when all my cards/banks started offering options like this. I essentially never have to go over my statements and transaction history with a fine-toothed comb anymore, because it's so so so easy to notice a fraudulent transaction immediately as it happens.

[ajb]

Can you even do you do that on AWS? Best I could find for the total spend is the cloudwatch EstimatedCharge metric, which only updates once a day (at least last time I checked)

Apart from that you seem to be limited to monitoring individual resources, or using some external service that enunerates everything

[cmeacham98]

It's updated only 3 times a day, but I would read this: https://docs.aws.amazon.com/cost-management/latest/userguide...

Disclaimer: I work on AWS, although nowhere near billing

[vb-8448]

If a hacker gets control of your account, probably the first thing he will do is disabling all checks. Does Azure(or even AWS/GCP) provide some "non-configurable" default alert?

[justsomehnguy]

Please, demand what you can have a hard limit on spending for the service/account or just a banal pre-pay.

It's amazing what I can have a pre-paid account for a VPS hosting in Nicaragua, yet Amazon doesn't have this as an option.

/rant

[janosdebugs]

What is AWS/Azure supposed to do if you run out of funds? Delete your resources? Storage costs money too, simply shutting down the services won't quite cut it. VPS consumption is relatively easy to predict, complex cloud services are not.

[justsomehnguy]

Why any VPS provider doesn't delete my VPS outright, instead nagging me for the payment?

Why any e-mail provider doesn't delete my account if the payment weren't received in time?

Why AWS/Az should outright delete my services as soon as the money on my pre-paid account hit $0?

> Storage costs money too, simply shutting down the services won't quite cut it.

Until you give AWS/Az $B in storage costs it costs nothing for them to store your measly couple of TBs of data for a month, till you sort out your money situation.

And to give you a perspective on the situation: I have a service in AWS. Every month AWS sends me tons of letters of how it can't charge me for the service, how this can lead to a service cancelling, what should immediately run and punch in a working card to the account or else.

The thing is what the cost of the service is ... $0.51/month. For $6 I could've had a year of nag-free service, for $60 I could've had a 10 years of nag-free service. I would happily thew not even $60, but $100 to AWS so I could just forget it about it, instead they spend more resources trying to charge me every month and sending me all those scary letters.

> complex cloud services are not.

There is absolutely nothing what would prevent something to alert you (or disable, if you wish so!) if your current spending is 5x or 10x of your previous month. In OP's situation this is 66x difference.

[matkoniecz]

It can be configurable. For my use I would be fine with terminating all activities and use reserve funds to pay for data storage.

[igorkraw]

Ask the customer to pay collateral when enabling this for a service, then use that to pay for the data storage in the shutdown grace period, reimburse it (or count it against future use) if it was never used. You can also easily add this on all the elastic services,e.g. your price per GB simply goes up a bit for S3 in the first month, until the amount of data you have is fully in your "insurance".

Or, you know, price it in and do your own risk estimates to make this a seemless experience, but AWS got away with not offering anything, so I'm already assuming you want to be passing all the cost to the customer (heck, amazon, if you are reading this, you can even skim a little bit of the "insurance" money! )

[popotamonga]

same thing it does when you dont pay. all goes down. databases readonly mode. Eventually it all gets deleted if you dont pay for a another month or so.

[SoftTalker]

Don't pay it. Send them notice, by registered letter, that the charges are fradulent. If a credit card was charged, try to initiate a chargeback/fraud claim.

Once you pay it, you lose all leverage. You're much less likely to ever get any money back.

Probably consult with a lawyer.

Cloud hosting charges are basically all profit for the hosting company. They didn't really lose anything except a bit of electricity. In my experience, companies are pretty willing to forgive fraudulent charges if you don't have an unusual history of them.

[dusted]

Playing the devils advocate here (though, to be honest, I am very much a hosted-on-premise kind of angry old man here)..

Allowing that is a slippery slope for a cloud host. If people can simply say "oh, someone used our credentials to do that thing that cost a lot of money" as a get-out-of-bill card..

If they were legitimately hacked, as in, the intruder did NOT simply obtain their access credentials, but actually bypassed the security system itself (hacking into the actual azure host, or exploiting a technical glitch in the azure login system) then, of course they should forgive the bill (and apologize to their customers)..

[qolop]

It's definitely not a slippery slope. Firstly it's a drop in the ocean for cloud providers. Secondly, there's some effort required in establishing whether it was a case of stolen credentials or not. Not everyone is going to put in that effort to lie. And finally, you're forgetting that most cloud users are enterprises and small business that have some moral compass and aren't going to lie to the cloud providers to their face.

[Salgat]

If your business is dependent on Azure, what happens when they shut down your services for lack of payment? This seems extremely risky.

[Rastonbury]

They business might not be able to pay 100x their monthly cost either

[kelnos]

If choosing to go this route, back up any data stored on Azure, and start looking into how to migrate everything to AWS or GCP or something else, without incurring too much downtime. Refusing to pay the charge, issuing a chargeback, or even getting a lawyer involved could get OP's account terminated, or at least suspended.

[amerkhalid]

Or even go back to good old VPS/metal servers. Maybe a little more work upfront but less likely to have surprises like this. Also more competition in that space and no worries about vendor lock-ins.

[kureikain]

Try to contact Microsoft support immediately.

Don't rely on the distributor/vendor, they act very slowly.

You're a customer of Azure, you can contact them by any mean, the fact you pay through a distributor doesn't change that relationship.

So I would open a Azure support, and also will try to find Azure team on Twitter/Hacker News etc and contact them politely for help.

There is no way you would have to pay this bill. They will sort out something or even waived it if it's the first time.

[ivanchaz]

That's unfortunate situation. It happened to me once before (though, we was using AWS that time. And, I believe the cost was smaller than the one you have right now).

What we did to recover the cost was to contact the account manager for our region at the time. So, maybe you could have better luck trying to find the particular person in linkedin. Or, have you tried opened a ticket from Azure console?

Nonetheless, I hope after everything has been settled down, you won't fire anyone (and treat it as learning opportunity)

[just-tom]

I'm quite surprised that there isn't some kind of monthly budget control. For every new project I set the budget to be 4-5x my expected expense.

[nurettin]

I asked microsoft support to vaive the past two months of billing because I left open a database cluster which I created for testing purposes. They promptly replied, took me through the steps and vaived the bills.

So maybe just file a support ticket, or have your distributor file a ticket for you?

[xwolfi]

When that happened to us, we found an article showing Tesla got hacked the same week as us (was aws) and they got the money back, so why not us?

We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone.

[onion2k]

It's a shame you failed to learn a lesson about how security is something the entire team is responsible for, and how a mistake like deploying something without a password is a failure of your processes rather than a failure of any one person. By firing the guy all you've done is made everyone paranoid which will slow you down; similar mistakes are still just as likely to happen again.

[ayewo]

You say: "We got the money back and fired the guy who had a jenkins opened without password, granting terminal access to anyone."

Why did you fire the employee (?) so quickly? Did he have a history of negligence and/or incompetence on the job and this was the last straw that broke the camel's back?

[jve]

Well, a guy who learned a thing or two the hard way and won't do that ever again got fired.

[oasisbob]

Why was that guy running Jenkins without a password?

[onion2k]

Years ago when I actually did any devops running services without a password was common. They would configured to only be accessible using SSH from a locked down IP range. It's far more secure than password based access (if you get it right).

[988747]

Yes, but that was a design/security choice. What OP described seems like a pure negligence ("setting up authentication is hard, so let's skip it").

[calvinmorrison]

I'm not sure why you would be liable for fraud

[gigel82]

To play the devil's advocate, why would Azure be liable for OP's security lapse?

This would've been much easier if someone stole your credit card and bought things with it (the CC company would help with the chargeback).

[causi]

Because it wasn't OP who was hacked? The victim of the crime is Azure, not OP. If I have an Xbox account and someone hacks it and buys a bunch of games, it is the criminal who is deceiving Microsoft into thinking they are someone else. Microsoft trying to charge me for something someone else did would just be a second incidence of fraud. I could leave my Amazon account open on my desk and, assuming I could prove it with security camera footage, someone could walk up and order something and it still would be them defrauding Amazon, not them defrauding me.

[cmeacham98]

I'm not a lawyer, so this is not a legal prescription (I do not know who is legally liable in this scenario, I suspect it depends a lot on the details).

That said, it seems like for society to work as it does we need people to take some level of responsibility over their action and inaction related to account security. If I live in the world you describe all online services will be forced to make you upload a photo ID for each purchase to confirm it is you.

[kelnos]

The problem with this stance is that the corporation naturally has much more power in the economic relationship than the customer does. If you give the vendor too much leeway to say "the customer should have been more careful with their credentials!" then they will always say that -- and usually prevail in that opinion -- even when the customer couldn't reasonably have done better.

[cmeacham98]

You seem to believe I said something like "we should always believe the company no matter what the evidence says", but if you reread my comment you'll find that I didn't.

[calvinmorrison]

Azure does bills on credit, IE: you spend and pay later. That's up to them, but it's far riskier than prepurchased credits.

I'd find a jury unwilling to believe that a similar real life scenario would raise no flags. It's only a flag raiser because tech companies have automated away all human interaction with billing. Imagine someone claiming to be bob, who regularly shops at the grocery store for 100 dollars a week, now wants to come in and spend, say, 10,000 dollars, on credit. This would be a red flag to any proprietor. Now imagine that proprietor going after bob, who was not there, and claim he is responsible.

[gigel82]

We don't know enough about what happened but I disagree this should be an automatic red flag from the provider's side. I'm sure Azure has spending limits and alarms that one can set up (and probably should).

The attitude of "provider should eat the cost" is ripe for abuse. I can set up some expensive GPU instances to train my GPT4 clone (millions worth of compute) than -after getting my models- claim I was "hacked" and refuse to pay my bill. Or maybe -more benign- have some buyer remorse after setting up a public instance for people to test out my new AI product then get scared when it gets the HN "hug of death" and my bill skyrockets.

[rtev]

This is 100% OP’s fault, it’s not like Azure itself was breached. Their account got hacked due to weak security practices on their part.

[bloppe]

Assuming the perpetrators are never caught and there is no insurance, then this isn't about liability. It's just theft.

[teeray]

I wonder if you can take out an insurance policy against this. Many of them have cyber-fraud coverage… perhaps this would qualify.

[bearjaws]

AWS would reimburse this if it was the first time. Maybe some hope for MS to do the same?

[benjaminwootton]

I had an incident where a SaaS product went haywire and ran up a $10k bill. I was quite shocked when AWS didn’t write it off and pursued me for the money. You can’t necessarily assume a cloud provider will have your back in these situations.

[schnebbau]

I accidentally ordered a $200 bottle of wine instead of a $20 bottle at a restaurant once, but didn't realise until after the whole bottle was gone. I was not shocked when the restaurant expected me to pay for it, because obviously I was totally responsible.

Why would AWS be any different?

[BartjeD]

... Because building and running a piece of software is a vastly more complicated process than ordering a bottle of wine and drinking it.

The way to hell is paved with well intended analogies.

[schnebbau]

The complexity of the process of causing loss to another does not absolve you from it.

[benjaminwootton]

It certainly mitigates it. I have many years of experience with AWS and installed Databricks, an off the shelf product through their SaaS portal.

I certainly feel less responsible and more aggrieved when that situation costs me a lot of money than in the $200 wine example.

[kelnos]

That doesn't really matter, though. A cloud provider would likely have different motives and apply a different calculus to the decision of whether or not to offer a refund in a case like this, than a restaurant would over a bottle of expensive wine.

Amazon is thinking about customer retention and growth, and about the goodwill lost in refusing such a request. The restaurant is thinking about the likelihood that someone is just trying to get out of paying for something for which they did actually understand the cost.

[kelnos]

Because the actual cost to AWS for a customer accidentally racking up an expensive bill is minuscule compared to the cost of that bottle of wine to the restaurant.

Amazon also has an interest in keeping their customers happy, and keeping usage growing. A customer who gets a refund for accidental or fraudulent usage is much more likely to remain a loyal customer, and hopefully spend more on the platform in the future. A restaurant, while in the hospitality industry, is likely more concerned with their razor-thin margins.

Also I would expect the true motive in your example is that the restaurant-goer is trying to scam the restaurant out of an expensive bottle of wine, and I'm sure most restaurant managers would agree with me. The AWS customer is not only more likely to have made an honest mistake, but AWS support has tools available to look at the usage and make a more nuanced decision as to whether or not the customer is telling the truth.

(And I have witnessed quite a few situations where Amazon has written off usage bills in cases like this. So clearly they agree with me on this, at least some of the time. Not all of the time, of course, as the grandparent poster can attest to.)

[chrisandchris]

Everything you say can be vice-versa said and will apply too (or if it does not apply, it's a subjective IMHO).

A restaurant has an interest in keeping customers happy, and keep recurring customers growing.

I would also expect, that the average cloud user made a mistake a professional should not have made and is now trying to get out of it the easy way. I mean, what do you expect if you post your credentials to GitHub (just an assumption)? Same as if you order wine without looking at the price. Your fault.

Also, a single bottle of 200$ is minuscule compared to the monthly/yearly renting fee a restaurant needs to pay, which can be many thousands.

Long story short: IMHO it all comes down to how good one can tell that it's really a mistake and there were multiple mistakes which could hardly be avoided which lead to this issue.

[voakbasda]

Some restaurants pull a bait and switch on customers ordering wine, bringing a similar bu wildly more expensive bottle in place of the one that was ordered, hoping for exactly the outcome that you described.

I would be shocked to find a cloud provider doing that, but I wanted to point out that you might be not have made a real mistake.

[benjaminwootton]

My point was not to assume that cloud providers would write off the cost of errors or security issues.

[onion2k]

I know of one startup that failed due to a bill not being forgiven after they were attacked. There is no guaranteed safety net with cloud costs. This is why it'd be good if you could put a hard limit on your account usage.

[Lorin]

Note this is only for 'real' money - if you are under one of their service credit programs that credit is gone like the wind.

[nickcox]

That was exactly my experience with an Azure overspend at a previous startup. The real cost was forgiven but the credits weren't.

[shrikant]

GCP forgave my team's bill of approx £220,000 when we'd misconfigured an ETL job that ended up running horribly expensive pay-as-you-go queries for about 3 days before we noticed.

It was pretty much entirely our fault, and we were still able to get those charges forgiven when we owned up to the error and asked nicely.

So I'd at least recommend asking Azure politely first.

[mkl95]

I can't help you with the legal side of things, but moving forward I advise hiring some security-aware infra guy. The root cause of most of these incidents is some human being incompetent (leading to things like poor security and relying on manual processes) or reckless.

[m4jor]

You had no 2FA enabled?