[orf]

I mean, it’s hard to have sympathy. Your setup sounds like a complete mess especially if you didn’t notice it. You didn’t even have MFA and you’re using the console to create resources?

It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t seem to have any infrastructure monitoring?

I’d honestly hire some people that have experience in this area, because your “dev team” sounds clueless.

[asdfthrowawayyy]

Nonsense, AWS is an industry leader in use of the asterisk. With exception to perhaps S3 and AWS Lambda, the predominant majority of AWS services are cloaked with accounting legalese and hidden billing gotchas that can easily result in a several thousand dollar bill within days if you don't analyze every aspect of AWS service offerings including their EULAs and acceptable use policies.

So yes, it was bad form for OP to not establish billing alerts, but you can't in any way say that AWS is transparent about most aspects of their service tiers.

[OmegaPG]

It is not in AWS interest to give too many options to their users to restrict the usage and expenditure.

AWS can certainly do a better job in telling me how to optimise my AWS hardware and expenses

In OPs case, if you search in Google, you will find this AWS hacking happen a lot with many users including me and AWS support was extremely kind to me to give additional credits to offset that expense.

[mattmanser]

If AWS hacking happens a lot it sounds like a problem for AWS to deal with by default, doesn't it? Perhaps even be forced to by law?

It would be trivial for AWS to force you to setup spending limits when setting up an account, one for an alert, one for a hard lock.

AWS billing is terrible too, even now I'm getting charged a few dollars on my personal account for who knows what, I've cancelled everything I can find. It's like whack-a-mole everytime you spin something up.

In the end that's basically theft by AWS but you can't make too much noise or your account gets cancelled.

[orf]

I mean, yes, AWS isn’t going to not take your money if you’re using their service. And no, it’s not some murky snake pit. There are some nuances but on the whole it’s pretty clear.

Anyway I was talking more about cloudwatch monitoring, including tracing all API calls. The billing is secondary: someone has been in his account and might have exfiltrated everything.

[csakon]

I agree that it should have been noticed earlier from a few vantage points. I'm managing over a dozen contractors which varies from dev to marketing to customer support, operations, sales, financials, investor relations, etc. Everything could be better, but it's a business and the squeaky wheel gets the oil. I never fathomed the squeaky wheel was going to be AWS usage charges.

I myself have never created an instance, I set up the account then gave access to the devs. I only log into the account to provide new access to devs that need it and none of them are full time.

Ultimately the responsibility lies with me, but I would disagree that my dev team is clueless. Rather they're working on development, not watching what the servers are doing on a daily basis so I think that's a bit unfair.

[orf]

Let me get this right: you give a parade of short-term contractors access to your production AWS account, presumably without proper permission segmentation, and neglect to do anything _other_ than that?

I assume you revoke access later, but I doubt you audit anything that they may have done (like create keys that outlive their access) in the account or that any of it is version controlled or traceable.

And you’re surprised you’re in this situation?

And fair enough, you pay your contractors to do a specific job. None of them are going to point out that the way you’re managing your infrastructure is pretty slow and inefficient, or that perhaps there’s a better way to do any of what you’re doing on AWS that is cheaper, faster, more secure and that might give you a far quicker iteration time with the added advantage that you won’t fall apart with a surprise bill like this. They, after all, are working on “development”.

[asdfthrowawayyy]

Economy of scale is an indispensable aspect of any competent engineering effort. Any reasonably adept AWS engineer would always include cost considerations for any change to AWS backend architecture, including helping the client decide which service tier makes the most sense for a particular application's use case...