[lacker]

You really need to implement limits service-by-service, and with as many services as AWS has, it would make sense if some of them hadn't implemented limits yet.

Think about the details... if you are paying something for both storage and bandwidth, and get a sudden surge of bandwidth, do you really want items in storage to be deleted? You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount; limits on its maximum size and alerts are much better.

But once you realize that bill capping doesn't make sense for storage, well, many different services are essentially some type of storage. This is a feature that sounds good but in practice what people really need is something slightly different.

[akira2501]

> But once you realize that bill capping doesn't make sense for storage

I'd think you want something like a circuit breaker.. steady costs or slowly increasing costs aren't like to be a problem, but a sudden surge in costs is what I'm concerned with.

Perhaps if it operated like a time-averaged quota.. don't let me incur _new_ costs if I slide too far out of my apparent range. Give me a knob to control that range or temporarily disable it, and maybe a way to monitor those events so I can react to them appropriately for my particular application.

[p1necone]

I don't think you ever want storage to be deleted as part of an automatic bill cap system. Just refuse access, or allow reads but not writes until the customer reviews it. There's even an HTTP code for it: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402 (non standard, but I know for a fact it's used by Azure).

Actually deleting things should be done after a period of time specified in writing in the contract with the customer, and I'd hope you'd try to contact them first.

But I think an automatic bill cap absolutely makes sense for storage, and I can't really see any reasons it couldn't work.

[nucleardog]

> and I can't really see any reasons it couldn't work.

You’re assuming the existence of a function which can extrapolate the daily/monthly/etc costs.

If I upload a 10GB file at $0.10/gbmo, how much will it cost me? We have no idea because we don’t know how long it will be up there.

We have a process that downloads and streams massive reports into S3, then a process is immediately kicked off to handle processing and importing. As soon as that completes, the files are deleted.

So we’re paying about 0.04 months of storage every month. The naive solution is estimating 1.00. You’re off by a factor of ~24.

We could... not handle that. Now we’re in a situation where we’re consistently spending $100/mo on storage, set a liberal $150 budget and immediately everything breaks. We actually needed to set a $2400 budget! Expecting a $100 bill and getting a $2400 bill some month when something goes wrong is as good as doing nothing.

You’re also going to need to account for the interaction of every single object in a bucket and the lifecycle rules set which is an absolutely ridiculous amount of overhead.

Now extend this to... literally every one of the seemingly bazillion AWS services.

The only cap that’s going to be easy and realistic to implement without a crystal ball is a hard cut-off once you’ve actually accrued the charges, and a “shut down all my servers, delete all my data, and just close my account” cap is useful to almost no one.

If you want useful caps they need to be aware of your workload. The best person to implement those is you. The tools are there.

[p1necone]

You're over complicating things, I'm not assuming the existence of any magical function (other than the one Amazon already uses to calculate your bill). All I want is to be able to say "I do not, under any circumstances want to spend more than $X per month on this service." Amazon just needs to keep track of usage, and start rate limiting/returning 402s/blocking writes when the monthly limit is hit.

If I estimate I'm going to need to spend $400 a month on my bill, then I might set a limit of $800, send me an email warning when I'm getting close, I'll go in and work out why I'm consuming much more than I expected. If I don't hit the "let me exceed my billing limit this month" button then stop the service. (but don't delete any data, I'm not sure why so many commenters seem to think this is a necessary step).

Amazon is capable of calculating how much to charge you every month, so they're capable of doing this too. No fancy estimation required.

==================================

So under your example - I upload my 10gb file, but some bug in my processing code means it doesn't get deleted and sits there for a while. Maybe I don't notice and it sits there for a few days instead of a few seconds, but Amazon sends me an automatic email because my usage this month so far is X% higher than the average of my last Y months.

Ideal scenario: I go in, investigate, delete the file and temporarily increase my billing cap just for this month.

Not great, but okay scenario: I don't notice the email, a few days later and I'm at ~2x my previous months bill (or whatever limit I've set up), amazon automatically starts returning 402s when I try to access my storage service. I am happy because this is still much better than getting a bill for 10x my previous amount at the end of the month instead.

[nucleardog]

> Amazon just needs to keep track of usage, and start rate limiting/returning 402s/blocking writes when the monthly limit is hit.

You're accruing costs even if you stop reading and writing. You're paying for the on-going storage, running of instances, etc.

What you're describing does not implement what you're describing it as.

E.g.,

It's the first day of the billing cycle, so our bill is effectively $0. So no limits in place on writing/rates/etc. We upload 100TB of data. No rate limiting or blocking because our current bill is under the $800 limit.

By about the 8th of the month, we've now hit the $800 limit you set. If you don't go hit the "let me exceed my billing limit" then you want Amazon to "stop the service".

You want:

  1. Under no circumstances to spend more than $800/mo.
  2. Your data to be retained.

This is not possible. There are two paths forward here:

  * Amazon retains your data: You need to continue paying for storage. Your bill at the end of the month will be ~$2200. Fails #1.
  * Amazon "stops the service" (storing your data) and under no circumstances exceeds $800/mo in charges: They delete your data. Fails #2.

You're asking for Amazon to provide you services and just not bill you for them if you don't want to pay for them. This isn't going to happen.

Alternatively, if you want them to not accept a write if it _would_ have led to you being over your budget, see my original comment about predicting the future.

[lupire]

You're twisting things. Amazon can alert you of the storage use and threaten to charge you in a few days, not a month. They can give you grace once per year so you're "on notice" after your first goof.

[nucleardog]

They... already do that.

Billing alerts take two seconds to set up in CloudWatch.

If it's a legitimate goof, contact support and they've refunded me every time.

Glad we're all on the same page then and there's no problem here.

Also interested to hear exactly what part of what I said was "twisting" or misrepresenting the issue.

[p1necone]

AWS doesn't run on a magical black box that eats customer money and spits out storage. They absolutely can eat some costs in a minority of cases in order to provide better service to customers (and I'm sure they already do in many other ways).

[nucleardog]

They are already quite willing to eat costs in cases of legitimate mistakes, compromised credentials, etc for the sake of customer service. I think I've worked with organizations at this point to have a couple hundred thousand dollars worth of bills refunded. The bulk of that was a single $110kUSD bill.

They provide billing estimates and billing data through CloudWatch metrics. This can be used to alert via SMS, email, and other methods. These can be used to trigger lambda functions to implement your own "shut it down" functionality quite easily in a way that actually makes sense for your workload.

What you seem to be advocating for is, more or less, a "pay what you want" model. If they're going to provide services and let you choose the maximum you're willing to pay and expect them to eat the rest, then I don't know how else to describe it.

[capableweb]

> You basically never want storage to be automatically deleted even if your program suddenly uses a surprising amount

In most use cases you're right. But never? Not true, not all storage is used the same way. And the thing is, why should Amazon decide which use case is valid or not? I might care more about being able to afford the service than to keep the data around, depending on what kind of service I offer my user. As a platform provider, they should be agnostic, but their greed for money is shining through their willingness to be a true platform provider.

[philliphaydon]

There are rules in S3 to move files or delete them after a period of time.

[Hamuko]

And perhaps most useful rule is to move them to cheaper tiers of storage. For example, Glacier Deep Archive is super cheap for storage since it's $0.99 per terabyte-month.

[forty]

Exactly. I have heard stories of bill capping that ended up with infra and data being deleted, so I can really see why they wouldn't implement it. What they could do however would be to allow usage limit based on IAM policies for example (Deny s3:Getobject Condition: monthly bandwidth >= x)