[hermitdev]

I spent an entire summer as an intern, constantly on the phone with IBM support trying to get DB2 Connect to work in a Windows cluster only to be told "this is not a supported configuration" despite it being clearly supported in the documentation. Gave up. Years later, ran into my former boss randomly (at a liquor store of all places) and he mentioned they were finally able to get it to work.

Another confounding one was where I was trying to bulk copy some data to Sybase using Python. Started getting some really strange DB errors. Couldn't figure it out for a while. Turns out, it was a bug in the DB module and it was using uninitialized memory in certain conditions. Was a 1-line fix, but took about 6 weeks to find.

Yet another one was when I was working on porting my C++ services from 32 to 64-bit. Sockets were timing out immediately sometimes. Couldn't reproduce in the debugger. Was a bug in a 3rd party framework. It was improperly using the rtdsc instruction in some inline ASM. Worked fine on 32-bit, but the register layout was different on 64-bit. So, it was effectively reading a garbage upper 32-bits for the high-res timer. Only found that one because I noticed in my logs that my timers were reporting that some operations had taken >200 years. I forget how long it took to track that one down, but it was months of off and on hunting.

I've also hit internal compiler errors. The one I remember was that an anonymous namespace at global scope would cause an ICE. I was about to file a bug report once I'd a minimal reproduction, but it'd already been reported and fixed.