[deathanatos]

I found and eliminated a $20k/mo mistake once. It was … stunning. In an instant, I paid for myself, and to very little fanfare. Just poking around trying to understand the high level about where the costs were.

On the more mundane side, rewrote a script from bash to Python. The change didn't really change the script, in that the original is more less doing its job (just… poorly, and wasting a lot of human time on the side), but the output is much more readable, and should quell a lot of confusion from people not grokking exactly what the output of the script is telling them. (You really had to read the bash to understand the output. But the Python can pretty it up, by way of being able to parse & collate… so it should be much more understandable. Also fixes a few bugs from the original…)

[silicon2401]

Awesome job. Makes total sense how switching to Python would improve things. What was your approach to understanding what the costs were, like just looking at memory and space cost, or its cost when running on the actual infrastructure?

[deathanatos]

The cost thing was just looking at the bill. I've worked at some companies where engineering didn't even have access to the AWS bill, for example, which is rather unfortunate, since engineering is where the knowledge of "wow, this pie slice shouldn't exist" is. (I do think finance could? should? have noticed that bill, since I in part noticed it b/c the monthly total was too high. But I might have been looking at a more broken-out number than the actual, final total there were some other things that might have gotten mixed in that would have dulled the signal enough to make it plausibly missable.

But after seeing the number, it was a pretty quick task to break it out into various areas over time (like total VM$/time, total storage$/time), find the category that grew, and repeat until the offender was found.

The script thing was mostly about time. Bash's output was only so intelligible (it was basically |grep|awk|grep sort of thing) and engineers would get confused, ask questions, and it soaked up enough time to make me say "let's rewrite this so I never have to answer these questions again".