We noticed that one line of byte code was crazy hot. We traced it back to the code, and realized it was a special case fallback.
They were summing 10mb of 8 bit encoded integers, in the 99.5th percentile of traffic, so no one noticed. It was using 10% of fleet compute.
We changed the encoding to 64bit; the data size stayed constant,
the 99.5%tile latency plummeted, and I got a bonus. Took a week. My contribution to climate change.