[caseymarquis]

So, what are best practices to avoid this situation in the first place? MFA. Billing alerts when estimated charges are over expected spending amounts. Anything else? Seems like a small mistake here could really harm a small business. Are there good ways to detect access that hasn't yet been exploited? Someone mentioned monitoring API calls, but what I'd googled on that seems fairly broad.

[ryan29]

I've been looking into it and I think it would be possible to do if the documentation were better. They have `Budget Actions`, but it's fairly new so the documentation and examples are lacking. Based on what I've learned for myself, I would say the starter rules are:

    - Create a root account that's only used for consolidated billing and account recovery.  Secure it with 2FA.  I use TOTP saved on my Yubikey and my backup Yubikey with the setting that requires a physical touch to generate a code.
    - Create organizational accounts for every day use.  The exact way to structure them can get complicated, but there's a fair bit of documentation on it.
    - Set up budget alerts and budget actions before you start using a resource type.
    - Only create users with permissions to access resource types with budget actions set up.

It's easy to say, but very hard to do in practice based on my experience. The biggest problem is that if you want to use a couple of services (ex: EC2, S3), you have the complexity of 1000 services, IAM policies, etc. jammed in your face right at the start and it's almost impossible to figure out what permissions you need to do something.

It reminds me of SELinux where the permissions are difficult enough to deal with that you can write an audit log while performing an action and simply enable all the permissions that were logged.

The second biggest problem is that runaway billing is far worse for small users than for large users and big tech only cares about other big users because that's where the money is. Everything revolves around catering to huge users who don't care if they need to hire a consultant to tame their AWS billing, so the smaller users and startups are left with systems that are far too complex to meet their needs.

I prefer the way Digital Ocean works, but there are some things you just can't do with them. For example, Lambdas and SES don't have good alternatives at DO.

I also like Cloudflare Workers since I find it significantly easier to reason about price in the context of cost per execution instead of the complex formula used for Lambdas, etc.. I think Cloudflare is in a very good position to claw market share from the big clouds, but their Workers Unbound is pretty much a copy of Lambdas, Functions, etc. in terms of pricing structure, so it looks like they might be starting to go after those fat egress charges that everyone else makes their money from.

[isbvhodnvemrwvn]

Monitoring API calls is typically reactive on AWS - e.g. GuardDuty relies on logs from CloudTrail, which are incomplete (e.g. sending a message to SQS is not logged at all) and log delivery is delayed as well. Nevertheless it should detect the specific issue described here fairly quickly, certainly in less than 2 months.

Overall the best defense is defense in depth - use MFA for all human accounts, use IAM roles wherever possible, don't put stuff in public subnets, use restrictive firewall rules, follow least privilege principle, use secrets manager or similar services for storing credentials. You could write a book about it. Many people pretty much have.

[blibble]

create a subsidiary: YourCompany Web Services LLC with minimal capital (only used for AWS)