Hacker News new | ask | show | jobs
by hughrr 1739 days ago
Yeah. By the time you have woken up and read your billing alert email it’s too late.

However the reason it doesn’t exist I suspect is twofold. Firstly because it is bad business. All the cloud providers make a lot of money from mistakes and small things sapping cash. Secondly it’s hard to rationalise what to do when the budget runs out. What do they nuke?

4 comments

There's a third reason, which I suspect is the biggest. The billing is not real-time, and it's hard to make it even nearly so, especially in such a complex and heavily distributed infrastructure like AWS.
They have realtime insights into how much read/write capacity or IOPS happens on the majority of their services. Their throttling has millisecond resolution. They have no problem giving you realtime insight into when you need to push the magic button to spend more money and increase your write/read capacities. If billing isn't realtime, it's due to sheer laziness or malfeasance (plausible deniability, my guess), because the data is there.
I asked about this a few years ago. The answer I was given by various friends throughout AWS was consistently that of "oh, our customers don't want that" for whatever that's worth.
This is probably technically accurate. After all, I'm sure Amazon, as any business, weights customer voices by the amount of spend. The people spending millions don't want a hard stop (i.e., kill our production services when we hit a certain amount of spend); the only people who want it are the people spending comparatively small amounts, pre-revenue startups, individuals, etc.

This is an example where being data driven to the exclusion of all else can hurt a company; I suspect having this feature would pay dividends down the road (by being the first to provide a safety net for a startup with a fixed budget that doesn't have production workloads yet you offer a competitive advantage between cloud providers), but the effect is completely impossible to predict or track currently since it doesn't have an immediate impact on revenue or the satisfaction of large, paying customers.

While I hate it, I agree with this in that in most settings heads will roll if your main moneymaking service is offline because of a billing snag.
Pro-tip: "the data is there" doesn't mean it's cheap to use.

There's a very simple explanation for this; realtime billing would increase the cost of the product they sell to create something most people don't need.

If you can’t accurately tell someone how much they have spent, how can you expect them to stop before it’s too late?

If you can tell, then you can set a limit.

Besides, if they can trigger alerts at a particular spend then they should be able to create a limit.

>Besides, if they can trigger alerts at a particular spend then they should be able to create a limit.

That's not really true. The alerts happen when the billing is re-calculated (periodically) and you've exceed a predefined, not when you hit that exact threshold.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

>When you enable the monitoring of estimated charges for your AWS account, the estimated charges are calculated and sent several times daily to CloudWatch as metric data.

Real time billing is actually a Hard Problem to solve.

> Real time billing is actually a Hard Problem to solve.

I refuse to believe there is no workaround. I can understand it is not easy to fix for corporations who need AWS to make money but that is not the use case for students.

If it were, Azure for students couldn't exist. Signing up for Azure for students does not require a credit card so they must have figured out a way to prevent / stop the bleeding?

https://azure.microsoft.com/en-us/free/students/

Well stop the service on that periodic cycle then if it’s over the limit?

Non-realtime limits are better than no limits at all. Besides the cloudwatch documentation seems to suggest it’s reporting on a 5 minute frequency for most of AWS.

Billing is very tricky from the metering to the final PDF bill which includes taxes, promotions and whatnot. So this is hard if you want to put limits on a certain dollar amount. You could also put hard stops on resource usage (say, no more than 500 CPU/hr/month)
Then allow to set a pre tax pre promotion limit?

Besides, AWS already complicates things way too much by handling VAT like other billable items instead of just adding it at the final step like any sane company would

I think the solution would be to allow customers to set hard limits, so they at least have an upper bound on their monthly spend, while still being charged on their actual usage (in a nonrealtime fashion).

This also solves the problem of "what to cut". If I hit my bandwidth limit AWS simply stops routing requests to my servers, if I hit my CPU limit AWS should throttle me, etc.

This feels (no disrespect to you) like a huge cop out. How many more small, "let's hack this together and hope for the best" projects are they missing out on because developers feel uncomfortable with the black box that is AWS billing? I suspect it would be a significant number, believe it or not.
Would it be that hard to build some cost overrun plans?

If threshold (x) hit then do:

- Email me

- Stop Servers XYZ

- Leave Servers ABC running.

If threshold (y) hit then do:

- Email me / Call me

- Shut everything down.

Shared hosting providers from the late 90s onward typically had systems like this standard.
Both AWS and shared hosting providers run on a “F you. Pay me!” model. The difference is that small hosting resellers knew they couldn’t collect on debts while Amazon knows it can.
If your talking about crappanel exceeded bandwidth suspension page it wasn't realltime, can't remember if default was 1hr,6hr, 24 though
Even if not realtime, it sounds like a better option than AWS' current "not supported at all; you'll have to manually shut things down if costs overrun".
I'm pretty sure they didn't have to potentially call as many clients as today. :)
AWS sorta has something like this already with CloudWatch, but it'd be nice if it was simpler and immediate instead of reactive. I just run a little reserved EC2 instance so my main billing risks are excessive data out or forgetting to renew a reservation and reverting to by-the-hour billing. So that I don't worry about it much I have three alarms that notify me if either "estimated charges > $x for 1 datapoints within 6 hours", or there's "anomalous" NetworkOut over a day, or "there's more than X total NetworkOut over a day", and another alarm that's "NetworkOut > Y for 1 datapoint within 15 minutes" that notifies me and shuts the instance down. I'd like to have a hard cap of "my instantaneous billing running total for the month, not my 'estimate', has exceeded $x, shut everything down" but what's there is something.
I use AWS for a very small side project. I used about $10 back in July, and I don't expect any more costs for several months (aside from S3 hosting).

It would be really nice if I could preload $100 into the account and remove my credit card. I don't have ANYTHING sensitive behind my username and password- except my CC #

I know they are never going to implement this because I'm small potatoes, but it would be nice

Do they say that you won't go over the prepaid amount? I can't see that in the pages that I've read, but maybe I missed it.
Bin checks will usually disallow prepaid cards to reply below
Buy a prepaid cc.

$7.00 for up to $500

For data out I have some alerts set similar to yours to shut down the instance if NetworkOut becomes unusually high—over three different time scales. Since the alerts are delayed, though, I also set up traffic shaping rules in the VM to throttle the NetworkOut to something reasonable so that it doesn't incur a huge bill before the first alert triggers. Officially the VMs have 5 Gbps network links; at that rate you can accumulate quite a large bill in a very short time if the VM starts saturating the link.
I came here to basically say something like this: if there's a hard limit, your business will basically shut down entirely. It's hard (impossible) for cloud providers to automagically shut down services based on each customer's priorities.

I don't think there's a magical solution to this. There might be a company that sets a $1K USD/month limit, forgets about it, and suddenly the cloud provider shuts down everything a year later, while "everyone" is unavailable or something like that.

There are so many scenarios, and I honestly feel that the cloud providers have decided on the most fool-proof solution both for them and their clients.

>Shut everything down

Ok. But the Auto Scaling Groups are free - so I can keep that on, right? Oh, look! They just launched more EV2s, how convenient. Should I back these up to S3? With CRR enabled?

Tee hee hee

No, it’s not hard to figure out what to do when the budget runs out. Preserve data and shut everything else down.

So switch off all VMs, but don’t delete the disks. Disable S3 read/write, but don’t delete the data. Etc…

Preserve data is also costly.
That scenario implies that someone has made a mistake. They forgot to turn off a service or turned on a service by accident. How does AWS know that the billing cutoff wasn't the accident? Maybe I accidently set a $20 cutoff during building my MVP instead of $200, but now that I have a paying customer I'm going to hit $100/month. AWS could disable my very first customer because I forgot to fix my billing setting.

Doing nothing is generally better from a legal liability point of view. The customer should be liable for turning services on and off.