[ultimape]

I routed out a couple thousand heisenbugs in an old system I was working on. Turned out the web based platform had no cache invalidation / cache-busting feature enabled, so any time an update would be published, the browsers didn't always get the changes.

The icing on the cake was that it was our app that was bad, and the 'fix' they implemented was a completely broken work-around.

It was SOP to instruct the clients to turn off browser caching. So the app was slow as well. Inevitably the Client's on-site IT would install a new desktop and forget to turn off browser caching. So you'd get these weird states were errors would occur randomly and depend entirely on if someone had changed or hadn't set that setting.

Years of cruft and chaotic deployment workflow meant there were easily 10,000 different places where the cache busting would need to be implemented. I figured out a way to fix this using nginx as a caching layer, and using features of a brand name Web Toolkit which we had already partly implemented without caching. Even had a test harness set up and way to catch bugs during a transitionary stage.

Shame there wasn't any specific tickets on this exact fix because otherwise I would still be working there. Sure, I was assigned a bunch of the random error ones and found out what was causing them, but apparently being assigned a ticket and fixing the problem isn't part of my job?!

The head of the web dev team (and my manager) quit shortly after I was hired. I now understand why.

Fixing bugs is impossible sometimes.