[dmbass]

One of the worst embedded bugs I ever had the displeasure of looking into was on a data streaming device developed by a contractor. One of the I/O processes would lockup after something like 6 or 7 days of continuously running without error. My boss and I (the only programmers) scoured the codebase for any possible condition that looked like it might maybe somehow could cause a lockup and added debugging outputs. We ran multiple devices in parallel in an attempt to increase the likely hood of the bug occurring.

The weeks kept rolling by without progress and there was more and more pressure to ship the thing. We ended up writing code that would detect the lockup and perform a soft reset. Luckily there were only a few instances where resetting would cause a perceivable problem but it still hurts to know that you shipped incorrect software.