[donavanm]

Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after.

Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.

[01284a7e]

No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators?

1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

[CobrastanJorji]

There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.

[skissane]

> Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.

If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).

[brookst]

Thanks, I wasn’t familiar with XTS.

Now I’m picturing the 3am end of quarter fire drill in finance when they discover the company has accounts receivable of fourteen billion $XTS and it’s appearing in the quarterlies.

[raverbashing]

You certainly don't want your service caring about a test currency.

Nobody is pricing their services in test currency

They are either pricing in $ or in "units" and then converting upstream

This is an even bigger foot gun than what's happening here

[skissane]

AWS supports billing in multiple currencies: https://repost.aws/knowledge-center/supported-aws-currencies

So how is it a "foot gun" to add XTS as an additional currency, for internal use only?

I presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate - the services themselves don't know about UAE dirham, but the billing system does. So XTS just becomes another exchange rate. You could even fix it as 1 XTS = 1 USD, although choosing a different exchange rate than parity is likely to surface more bugs.

[raverbashing]

> AWS supports billing in multiple currencies

Of course. I'm not saying that it doesn't.

Read it again

I'm saying, the conversion to (any) currency needs to be done downstream of the service (to a general billing service). The service needs to bill "credits".

> presume AWS sets its prices in USD, and then converts them to the other currencies using the relevant exchange rate

Yes that's what I wrote in the first post

[qeternity]

But these aren't the right services where the test should be, right?

There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".

It's the third service that multiplies these things that failed. Where are the tests for that?

[DrewADesign]

To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice.

I’m totally guessing though.

[lostlogin]

> human-or-LLM-driven error.

In a computer system, dont those categories cover pretty much everything except a meteor strike?

[DrewADesign]

What I meant to say is an error from manual account management input— not a human error via code.

[CodesInChaos]

Occasionally you have faulty hardware. Most commonly RAM.

[mulmen]

Not everything is an LLM boogeyman. Sometimes a bug is just a bug.

[DrewADesign]

> Not everything is an LLM boogeyman.

Indeed.

>> To me this sounds like a human-or-LLM-driven error.

In fact, half of what’s implicated in that very sentence is not an LLM.

[thayne]

> well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime

You don't have a non-production environment where you can have billing that doesn't involve real money?

[rtpg]

the cynical take is of course that the tests were written by Claude and not validated.

Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.

[brookst]

I love how AI means that all human-written code is now perfect, and any problems are onbviously a symptom of vibe coding. Another few years of hearing this and I may begin to believe I only wrote perfect code, when my current memory suggests otherwise.

[scott_w]

Nah, this is not uncommon in AWS. I came across a similar bug 10 years ago when playing with Kinesis and nearly had a heart attack at a $2m billing alert! Turns out it was just 5c

[akdev1l]

Not even tests but just some basic anomaly detection lol.

Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

[qurren]

You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.

You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).

[mightyham]

Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.

In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.

COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.

[eventualcomp]

Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org.

Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.

[dlenski]

This was very much my experience, having worked in two different sub-organizations at AWS, and on several different services, in two different countries.

There's just extreme variation in the quality of the management, the quality of the engineers, the operational/development role split, the on-call schedule, and the development and testing methodology.

[awakeasleep]

Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.

If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.

That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.

[qurren]

From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document.

From the perspective of people you manage, it's a very different picture.

We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.

So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.

The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.

I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.

[smallerize]

> > everything is driven with sticks instead of carrots

> this is not the case

> [describes all sticks and no carrots]

[dlenski]

> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening.

Absolutely not my experience at AWS.

All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.

Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.

[RandomThrow321]

Obviously experiences vary greatly, but I was on one of the largest AWS orgs and they were quite punitive. People would demand them for perceived slights, then assign reviewers that were tough or on their side. Many of my friends had different experiences, though.

[embedding-shape]

> Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again

And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?

[nullorempty]

This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption".

There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.

And no one is shooting for better than OK.

[hoppp]

Yeah but this also negates the argument that people gotta use it or they get left behind.

If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.

[tyre]

Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.

[mendigou]

I am former AWS and this is pretty accurate.

The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).

[akdev1l]

Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways

[zelphirkalt]

Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.

[tmpz22]

AWS has had this reputation for over a decade. Every former AWS (including poached not fired) has relayed to me a verson of this.

Every once in a while (~1:12) you get one that sounds like a Mormon missionary praising how its not that way and AWS is perfect.

[qurren]

Yes, I am a former AWS employee.

I got put on Focus because my "contributions were not coming through" to leadership.

[geodel]

"Former" seems to an important detail here.

[switchbak]

If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else.

Or in my case, actively ignore any and all recruiting from that sesspool.

[gleenn]

If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.

[thewebguyd]

> take the 3am on-call and be the hero instead

Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"

I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.

[qurren]

The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.

[rootsudo]

This, exactly 100%

[dice]

How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?

[thewebguyd]

Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.

[mcpherrinm]

While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills.

I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.

I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)

It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.

[oenton]

> there's definitely this sort of monitoring in place. Surely somebody was paged.

Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.

[dclowd9901]

Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.

[Groxx]

They've already got anomaly detection: their users.

[el1s7]

I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.

[01284a7e]

If only there was some way to get anomaly detection services [1] inside of AWS...

1: https://aws.amazon.com/what-is/anomaly-detection/

[andai]

>no tests?

Earlier this week my slopservant implemented several comprehensive changes to a codebase.

It also wrote extensive tests to verify the correctness of the changes.

A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless.

The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect.

I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!

Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes.

https://en.wikipedia.org/wiki/Normalization_of_deviance

[antihipocrat]

I'm struggling with this on a hobby project now. I'm torn between continuing the fast pace of development and taking a pause and checking the entire codebase.

The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!

[andai]

Yeah my logic was, we're doing so many refactors that it doesn't make sense for me to start memorizing how things work until things settle down a little.

The argument in favour of "read each delta" (aside from catching them doing stupid shit) is to keep your mental model synced. But I already didn't understand my own code before AI, so that's moot too!

[antonvs]

> if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!

This is a well-known issue in formal methods, often referred to as the "formalization gap". It's pretty much an intractable problem, i.e. all we can do is manage it, not solve it.

In fact this issue exists in many fields, often under the name "specification gap" or "specification problem". You'll find versions of that discussed inv economics, architecture (of buildings, not software), etc. In architecture and construction, a high percentage of mistakes that result in lawsuits turn out to be due not to construction, but to specifications that were incorrect in some way.

The bottom line is that there's no perfect formal/mathematical way to go from intent to proof.

[lionkor]

The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details.

Then it can write tests based on the interface, and that is usually a lot more solid.

[NBJack]

That's yet another band aid on a hemorrhaging patient. And it still assumes the agent(s) are capable of correct comprehension of the purpose.

[lionkor]

Yeah for sure it's a bandaid fix, but it does produce tests that fail if the interface doesn't match the implementation, more often than not doing this does (in my experience).

With a skill like this, which I have written, the LLM will end a session with "wrote 24 tests, 5 failing due to API contract mismatch" or something, and leave it at that. I can then decide if that's a bad interface or bad implementation.

[tovej]

The lesson is: write your own code, it's not that hard.

[rtpg]

write the tests yourself. Or at least the sketch of the test (to get filled in later). Make a commit of what you do so you can then look at the changes.

If you write the tests the agents have an easier time writing the code!

[nonameiguess]

This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feels like a similar situation, where I'm sure they "test" what they know to test, in the way they know, but compared to a bank, well, it wouldn't surprise me the least if no one at AWS ever even thought to ask banks how they handle things like this. Instead of testing that a process works the way you specified it, you need to make sure you even have the right specification in the first place by consulting with prior art and ensuring whatever you translate into software reflects the legal reality of the process you're trying to encode.

There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless.

This seems to happen quite a bit with software companies.

[spydum]

I have bad news if you think banks are the pinnacle of cyber security practices...

Maybe some rare few are, but by and large they generally stink.

They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.

[antonvs]

The difference is that at bigger banks at least, they tend to throw more money at the problem. They'll have defense in depth with e.g. DMZs, proxies for ingress and egress, locked down workstations and browsers, etc., and a different team for each one of those domains.

So while the actual "on the ground" picture may look suboptimal at any given point, overall it does make for security that in practice is much better than average.

This is reflected in the actual security compromise statistics. Your money in a bank is a lot safer than, say, your credit card deals on file with many retailers.

[picofarad]

Does panjandrum mean chaotic rocketry, flahy but worthless in the end?

[roadhero]

we invoice clients monthly. if we ever sent one a bill 2^30 times too high, I don't think "estimated charges, no action required" would save the relationship :D

[bas]

Who needs tests when you have vibes?

[deltaray3]

It's just like in Superman III

[SlightlyLeftPad]

It’s fractions of a penny Peter.

[vasco]

I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.

[yawaramin]

The problem with an anomalous AWS bill is that I don't know if it's because of a bug on their end or because of a goof on my end, eg did I accidentally leave on some giant compute for a month or something. With a billion in my bank account I at least know that I definitely didn't put it in there.

[znsnsksjiaja]

If I experience a bug when transferring money (switching a digit or some such) they’ll shrug their shoulders and say there is nothing they can do. No amount of proof will push them to increment and decrement some integers over there.

They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?

[simmerup]

I’d be concerned as it throws into doubt their entire accounting mechanism

[hsbauauvhabzb]

What could a banana possibly cost Michael, ten dollars?

[DonHopkins]

...in the off chance that one in 100,000 will pay their bill instead of dropping dead from a heart attack (perhaps because there's a heartless OpenClaw running their accounting department), it's all worth it.

[mvdtnz]

Why would you think there are "no tests"?

[27183]

We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.

[chasd00]

why would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.

[27183]

I'm not quite following your objection.. I'd expect a test that checks the multiplier is correct would detect orders of magnitude discrepancy. So if you're billing $x/byte you'd write a test for the billing thing that checks that, given y bytes, the bill is x*y.

[edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.

[soniclettuce]

I think they're saying that their understanding of the hypothetical error is that somebody more or less configured a price as 5¢/byte instead of 5¢/GB. And if that's the case, that's not really a thing testing will catch. 5¢/byte is as "valid" a price as 5¢/GB.

There's a lot of hypotheticals in there, so who knows if testing could have caught this specific problem. But I see what they're saying in principle.

[hsbauauvhabzb]

Why have a default at all? Explicit is better than implicit.

[nullorempty]

Technically, there could be a test. It could just be wrong!

[27183]

If a tree falls in the forest and nobody hears it...

[edit] Testing your tests, like testing your backups, is a good idea

[fragmede]

Yes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.

[8note]

missing canaries more likely?

insufficient tests that dont assert on the right things?

the existence of a test doesnt mean it catches the right thing

based on the description, id bet the COE action item will be to do a migration that enforces units are passed at the billing service level

theres no good reason for the billing service to make up its own units.

[27183]

Yeah types and APIs that are difficult to misuse are way better than any regression test, canary, or what have you. Systems that are correct by construction always beat "you're holding it wrong".

[Twirrim]

I wonder if AWS billing still uses CSV files for passing data around.

IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!

[anvuong]

That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.

Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

[Twirrim]

Sure, and I would expect the COE called out the operational aspects. No one should have been in a position to be able to trigger that bug trivially (I don't remember if it came from a service, or if they injected the metering record themselves somehow, or quite what. Way too long ago for that)

Most engineers with more than a few years of experience know better than to directly try a stunt like that in production, but they do often get there through painful personal experience. I just wish I hadn't been one of the many that got to suffer from that engineer choosing to learn at that particular moment!

[mrwh]

Kinda want to push back on that. It's not like engineers are separate from operations, not really. A lot of operations is just what engineers have been socialised to do.

[9dev]

Eh… If I can bring prod down simply by sending CSV with a comma in a field to a prod endpoint, whoever has come up with the most naive CSV parser possible running in prod is responsible for the outage.

[slowdog]

100% agree with this, seems weird to call an engineer “smart” for triggering an API that allows any input but can’t handle it. I also kind of doubt it was an engineer testing vs some much more subtle issue as billing is in line for everything.

If this was 10 years ago, this was also the era when any real customer could write any ARN (resource identifier) they wanted including commas due to no expected standardization across services. This could have been trivially triggered by a customer.

Just a matter of time if no one is escaping commas in billing files.

[berkes]

Give that engineer a raise and an official title of "chaos monkey".

On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.

Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.

Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.

A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.

---

¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.

[Twirrim]

I got known as a "Chaos Monkey" at AWS, and have sort of carried on that title. It's not exactly accurate, but tech stuff just breaks around me, and it's very, very rarely my fault. I'm pretty cautious and resist any urge to "I wonder what happens..." with anything that could possibly have an effect beyond me.

The "Chaos Monkey" effect at AWS was so pronounced you could literally see on our sev2 count graph when it was my on-call week, because I'd get dramatically more sev2s than any other engineer.

The service I worked for at AWS was amazingly stable and reliable. A large majority of the alarms that ever fired, fired because of problems with another service we depended on in some way. This billing thing is a good example. I was on-call when there was a major SQS outage in a region, through a DynamoDB outage, S3 outages, all sorts of stuff. We used to joke that it would probably be a net benefit for AWS if I wasn't on-call, just so my spooky-at-a-distance wouldn't happen.

I eventually lost my "most sev2s in a week" record toward the end of my time there when someone else was on-call and DynamoDB had a major outage. DynamoDB held all of our metadata so everything broke and every alarm we had fired over the space of a few hours.

[hotstickyballs]

CSV files are widely known to be used by the most frugal companies so of course it is.

[berkes]

I've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far.

Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt.

I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day.

Because CSV is the "interface" that will bring your system down if not properly designed.

[fragmede]

Have you seen what RDS costs‽ AWS couldn't possibly afford that!

[donavanm]

It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.

[Twirrim]

It's one of those "I hate it, but I get it" situations. Migrating metering over to a whole new format on AWS's scale is a scarily large undertaking.

[sscaryterry]

This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular).

Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.

[shermantanktop]

But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions?

Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.

[picofarad]

Yeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars.

I'm stressed thinking about it, like from a theoretical standpoint.

[insane_dreamer]

actually $17B isn't so bad because it's so obviously an error and not something you can fatfinger. if you expected it to be $17 and it was $17,000 you'd be much more worried as that looks like a plausible error on your part

[jhickok]

Yeah that's what I was thinking. And if someone sent me a bill for 17 billion I would just tell them I don't have it and never will, sorry.

[8note]

its so far out as to be obviously not real

a smaller error by say, just one or two orders of magnitude are much more believable as a reader

[creatonez]

Not everyone got a 10 figure bill estimates, some posts were mentioning 8 and 9 figure estimates

[sscaryterry]

I've seen people have panic attacks over much smaller amounts.

Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.

[dlenski]

> Services emit metering values that arent directly tied to prices.

Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both.

I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.

[nurettin]

This is why I always fail loud rather than pick a stupid default.

[mlinhares]

The last thing a billing system should have is defaults, this is such an insane thing.

[thewhitetulip]

Hey man, vibe coders gonna vine code. No wait. Loop enginering.

We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!

[merek]

Were there cases of finalized invoices and fund transfers while the error was in place?

The sudden loss of working capital while awaiting a refund would be extremely disruptive for even large businesses.

[DelightOne]

Are you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?

[mlitwiniuk]

What apology letters? Never got anything, not even a reply to my support tickets.

[crossroadsguy]

Had it been half a million dollars or something or say like a few hundred dollars?

[QuinnyPig]

Was this the day that gp3 EBS volumes came out by any chance?

[donavanm]

Nope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.

[pudgywalsh]

"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."

[AlotOfReading]

Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then.

Worst case I've found was off by 15 orders of magnitude.

[cyberax]

One of my personal rules is to ALWAYS suffix the variables and fields with unit names:

`timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`.

Unless the variable type already constrains it (e.g. Duration in Go).

[monkpit]

Until someone comes along and changes it without updating the name, or the value comes from a remote service and they change without you changing your name… etc.

[markus_zhang]

This also helps downstream a lot.

[27183]

It's not difficult to write regression tests that catch unit mistakes.

[gleenn]

One of the Mars landers famously failed due to unit conversion errors from metric to standard.

[blemasle]

I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)

[fc417fc802]

Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures.

That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.

[reddalo]

You just need to start teaching kids the metric system. Then when they'll grow up the switch will almost magically happen by itself.

[Dan_-]

I know you’re being snarky, but the US system is not “imperial” anyway. It’s properly “US Customary” but is often called “US Standard.”

[golem14]

Wasn't it (also?) the Ariane V flight in 1996? Oh, NVM, that was an overflow error.

[zapkyeskrill]

Degrees, very funny.

[anvuong]

Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha.

Units and datetime will always be the bane of any professions ...

[fragmede]

names and money movement also come to mind.

[dev_l1x_be]

Imagine a programming language that has physical measurement unit support so this could have never happened.

[yawaramin]

Like F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...

[dev_l1x_be]

Exactly. There are other languages that can do that or worst case you can implement it yourself (not sure how hard that is).