[scrollaway]

AWS support doesn't generally suck or behave the way you're describing without good reason, so I feel we're missing part of the story here. What are you leaving out?

Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for the bill. If it hadn't been that, it'd have been ransomware, you probably got lucky.

Now, to see if your situation can be improved: Put up some dollars and get business support. Make a clear and polite case, from the beginning. Ask for a refund but you don't have grounds to demand it; if they issue one, it's a gesture of good will. They probably will issue one if you haven't had to ask for that before, but it reflects badly on everybody that cryptominers weren't caught for two months.

And before you create that ticket, make some billing alerts so you can show AWS support that this won't happen again.

[BugsJustFindMe]

> AWS support doesn't generally suck ... without good reason

I don't know what "generally" means here or what you're basing the claim on, but I'm with an organization that pays a lot of money to AWS and they regularly ghost us after giving a wrong or incomplete "works for me"-style answer.

[atmosx]

Same here. For 10k / month I would expect proper support and answers, instead we get access to an amazing "confidential newsletter" about 124344 projects we don't care about, problems we report are put on the 10-year plan and answers are usually links to documentation about the product per se.

I have no idea why we're paying.

[tonfreed]

AWS support puts me into a circle of "here's the details"->"you forgot other details"->"here's other details"->"you also need details"->"those details are in my initial message" more than any other paid support I've had to deal with.

They don't suck once they actually start doing something, but they do seem to have KPIs which incentivise wasting my time.

[bsder]

> They probably will issue one if you haven't had to ask for that before, but it reflects badly on everybody that cryptominers weren't caught for two months.

As much as I'd like to agree with you, AWS makes controlling this WAY too stupidly difficult.

Out here in the real world, many of us are part of startups that have 4 people and a dog. We wear many hats, and "AWS Billing Expert" is not one we have time for.

I actually grind my teeth and recommend Azure to most small companies on this alone. GCP is nice, but I simply won't recommend them due to Google.

However, sometimes AWS has "that service" that you really need. And I just have to caution people that AWS will not protect you.

AWS could solve this. Simply allow people to opt into a hard stop on spending--some of us would rather be down than overspend. Let us make that choice.

The fact that AWS absolutely refuses to solve this speaks volumes.

[isbvhodnvemrwvn]

Having billing alerts is the bare minimum I'd expect someone half-competent relying on pay-per-use resources to set up.

Hard stop on spending would delete all your data and would not cover things which are billed e.g. monthly. There are AWS budget actions which address some of the issues (e.g. can put a hard deny on any actions or stop your EC2 instances), admittedly a relatively new service.

[csakon]

I don't think i'm leaving anything out. It was my account (which now has had password changes and MFA set up), but I don't understand how there weren't red flags on the Austrian IP address login and the sudden spike in usage. I realize (now) that CloudWatch exists, but not sure why this isn't standard.

I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking the first went through.

Once access was made, we were completely unaware of their existence until I saw the charges and asked our devs about it. They said they didn't have any knowledge or access to these new instances.

I appreciate the advice, will upgrade the support and try again.

[scrollaway]

AWS gives you access to a lot of footguns, and they expect you to implement proper security practices, access controls, monitoring, billing alerts, etc.

Why? Because they serve just about every use case and scenario. I've myself spun up $5k/day resources on accounts that did four magnitudes less / day before that. There are some limits by default which you can get raised, but they're not going to prevent a $50k bill - at best, they're here to prevent a $500k one.

Anyway I agree they could make it clearer that you have to do all this crap yourself but it makes for a poor sales pitch.

[csakon]

Footguns...lol. New term for me. I see your point and it makes sense from that perspective. For me as a small business owner, it's very painful considering I tried to go the normal route of opening a case for help and simply cannot get anyone to escalate this or talk to me on the phone.

[scrollaway]

I understand. It's a tough situation. Breathe, AWS is one of the better services to end up in such a situation with. This will get resolved.

Be a little patient, but don't try to superescalate by going through emails and what not. Just file a business ticket. If this needs escalation, they will do so.

[tonfreed]

AWS is like a weapons cache you've stumbled upon in the middle of the desert, lots of fun, useful and interesting stuff in it but you're going to get yourself hurt if you don't take proper precautions.

This sounds like a cautionary tale. I have spending alarms on my personal account for this very reason, I'll know within 5-10 minutes if my monthly spend is going to break $50 because I've set up my alarms.

Your other option is to start a Cloudtrail and alarm on foreign IPs that are logging in, new IAM users and keys being created and changes to any alarms you have in place to check for this stuff. It won't necessarily stop it, but you'll be able to react a lot faster.

[thorin]

You'll have a notice within 5-10 minutes if you continually carry your phone and are "supporting" your application 24/7. What if you want to go camping or turn your phone off when you go to bed or do a long drive or something?

[usr1106]

I don't want 5-10 minutes necessarily.

But what would be best practices for billing alarms? I have used them in the past when I used a bit more of AWS, but I don't think I ever got one (which is good).

But it happens to me that I miss emails that I would have liked to read in time for weeks. That could happen with a billing alarm, too.

Maybe you could forward it to SNS with SMS delivery. But as a matter of fact SMS is one of the few services (if not the only one) with a spending limit. If that is reached you silently won't get SMSes anymore, I have experienced that.

[tonfreed]

No need to be facetious, friend. Were I a more paranoid man I'd hook a lambda into the SNS topic and terminate all ec2 instances, delete all S3 buckets, delete all IAM keys and regenerate and send me the root password.

I'm not that worried though.

[mattmanser]

There's nothing facetious about his post, you've over-reacted to it.

[isbvhodnvemrwvn]

Are you going to know it within 5-10m? Billing updates can take hours to appear on my account (I noticed that with some new services I was using). Even CloudTrail can take longer than 5-10m.

[orf]

I mean, it’s hard to have sympathy. Your setup sounds like a complete mess especially if you didn’t notice it. You didn’t even have MFA and you’re using the console to create resources?

It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t seem to have any infrastructure monitoring?

I’d honestly hire some people that have experience in this area, because your “dev team” sounds clueless.

[asdfthrowawayyy]

Nonsense, AWS is an industry leader in use of the asterisk. With exception to perhaps S3 and AWS Lambda, the predominant majority of AWS services are cloaked with accounting legalese and hidden billing gotchas that can easily result in a several thousand dollar bill within days if you don't analyze every aspect of AWS service offerings including their EULAs and acceptable use policies.

So yes, it was bad form for OP to not establish billing alerts, but you can't in any way say that AWS is transparent about most aspects of their service tiers.

[OmegaPG]

It is not in AWS interest to give too many options to their users to restrict the usage and expenditure.

AWS can certainly do a better job in telling me how to optimise my AWS hardware and expenses

In OPs case, if you search in Google, you will find this AWS hacking happen a lot with many users including me and AWS support was extremely kind to me to give additional credits to offset that expense.

[mattmanser]

If AWS hacking happens a lot it sounds like a problem for AWS to deal with by default, doesn't it? Perhaps even be forced to by law?

It would be trivial for AWS to force you to setup spending limits when setting up an account, one for an alert, one for a hard lock.

AWS billing is terrible too, even now I'm getting charged a few dollars on my personal account for who knows what, I've cancelled everything I can find. It's like whack-a-mole everytime you spin something up.

In the end that's basically theft by AWS but you can't make too much noise or your account gets cancelled.

[orf]

I mean, yes, AWS isn’t going to not take your money if you’re using their service. And no, it’s not some murky snake pit. There are some nuances but on the whole it’s pretty clear.

Anyway I was talking more about cloudwatch monitoring, including tracing all API calls. The billing is secondary: someone has been in his account and might have exfiltrated everything.

[csakon]

I agree that it should have been noticed earlier from a few vantage points. I'm managing over a dozen contractors which varies from dev to marketing to customer support, operations, sales, financials, investor relations, etc. Everything could be better, but it's a business and the squeaky wheel gets the oil. I never fathomed the squeaky wheel was going to be AWS usage charges.

I myself have never created an instance, I set up the account then gave access to the devs. I only log into the account to provide new access to devs that need it and none of them are full time.

Ultimately the responsibility lies with me, but I would disagree that my dev team is clueless. Rather they're working on development, not watching what the servers are doing on a daily basis so I think that's a bit unfair.

[orf]

Let me get this right: you give a parade of short-term contractors access to your production AWS account, presumably without proper permission segmentation, and neglect to do anything _other_ than that?

I assume you revoke access later, but I doubt you audit anything that they may have done (like create keys that outlive their access) in the account or that any of it is version controlled or traceable.

And you’re surprised you’re in this situation?

And fair enough, you pay your contractors to do a specific job. None of them are going to point out that the way you’re managing your infrastructure is pretty slow and inefficient, or that perhaps there’s a better way to do any of what you’re doing on AWS that is cheaper, faster, more secure and that might give you a far quicker iteration time with the added advantage that you won’t fall apart with a surprise bill like this. They, after all, are working on “development”.

[asdfthrowawayyy]

Economy of scale is an indispensable aspect of any competent engineering effort. Any reasonably adept AWS engineer would always include cost considerations for any change to AWS backend architecture, including helping the client decide which service tier makes the most sense for a particular application's use case...

[xtracto]

> I realize (now) that CloudWatch exists, but not sure why this isn't standard.

CloudWatch is standard AWS, or what do you mean?

Amazon gives you all the tools to monitor your usage and management of the cloud. You did not use those tools, had sloppy security and now it is Amazon's fault?

Take it as an expensive lesson learned. And get someone who knows what they are doing in AWS to administer it. Maybe your company was pennywise pound foolish by not having someone who knew AWS?

[asdfthrowawayyy]

AWS support completely sucks unless you are paying 10K+ per month for the enterprise support tier.

My average ticket response with paid AWS Developer is probably 72+ hours, and that's just for the initial triage what's up query.

Unless you are either a seasoned Linux developer with many years of Linux internals experience, or you have an enterprise level account, all of the lower class AWS support rungs are largely meaningless.

[oerpli]

I work at a company with enterprise support and still found their contact persons & support completely useless.

We had a few training and introduction sessions with their people and it usually ended with us tuning out/joking about their useless slides and presentations (with "inspiring" Jeff Bezos quotes and brags about the number of packages ordered on Amazon.com that have 0 relevance for anything our company would like from AWS) in a private channel.

[scrollaway]

Support being slow and support being bad are two different things.

Billing team is separate anyway. You get a better "treatment" if you're paying (as in, they will look at your case closer), but you have access to the same service regardless.

[atatatat]

Not if your business makes money everyday.