[Twirrim]

I wonder if AWS billing still uses CSV files for passing data around.

IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!

[anvuong]

That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.

Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

[Twirrim]

Sure, and I would expect the COE called out the operational aspects. No one should have been in a position to be able to trigger that bug trivially (I don't remember if it came from a service, or if they injected the metering record themselves somehow, or quite what. Way too long ago for that)

Most engineers with more than a few years of experience know better than to directly try a stunt like that in production, but they do often get there through painful personal experience. I just wish I hadn't been one of the many that got to suffer from that engineer choosing to learn at that particular moment!

[mrwh]

Kinda want to push back on that. It's not like engineers are separate from operations, not really. A lot of operations is just what engineers have been socialised to do.

[9dev]

Eh… If I can bring prod down simply by sending CSV with a comma in a field to a prod endpoint, whoever has come up with the most naive CSV parser possible running in prod is responsible for the outage.

[slowdog]

100% agree with this, seems weird to call an engineer “smart” for triggering an API that allows any input but can’t handle it. I also kind of doubt it was an engineer testing vs some much more subtle issue as billing is in line for everything.

If this was 10 years ago, this was also the era when any real customer could write any ARN (resource identifier) they wanted including commas due to no expected standardization across services. This could have been trivially triggered by a customer.

Just a matter of time if no one is escaping commas in billing files.

[berkes]

Give that engineer a raise and an official title of "chaos monkey".

On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.

Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.

Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.

A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.

---

¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.

[Twirrim]

I got known as a "Chaos Monkey" at AWS, and have sort of carried on that title. It's not exactly accurate, but tech stuff just breaks around me, and it's very, very rarely my fault. I'm pretty cautious and resist any urge to "I wonder what happens..." with anything that could possibly have an effect beyond me.

The "Chaos Monkey" effect at AWS was so pronounced you could literally see on our sev2 count graph when it was my on-call week, because I'd get dramatically more sev2s than any other engineer.

The service I worked for at AWS was amazingly stable and reliable. A large majority of the alarms that ever fired, fired because of problems with another service we depended on in some way. This billing thing is a good example. I was on-call when there was a major SQS outage in a region, through a DynamoDB outage, S3 outages, all sorts of stuff. We used to joke that it would probably be a net benefit for AWS if I wasn't on-call, just so my spooky-at-a-distance wouldn't happen.

I eventually lost my "most sev2s in a week" record toward the end of my time there when someone else was on-call and DynamoDB had a major outage. DynamoDB held all of our metadata so everything broke and every alarm we had fired over the space of a few hours.

[hotstickyballs]

CSV files are widely known to be used by the most frugal companies so of course it is.

[berkes]

I've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far.

Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt.

I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day.

Because CSV is the "interface" that will bring your system down if not properly designed.

[fragmede]

Have you seen what RDS costs‽ AWS couldn't possibly afford that!

[donavanm]

It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.

[Twirrim]

It's one of those "I hate it, but I get it" situations. Migrating metering over to a whole new format on AWS's scale is a scarily large undertaking.