[mrsalt]

Can you expand on what do you mean by "just popped open the kernel"? I'm genuinely interested in this kind of "tribal knowledge" you talk about. Maybe there is something we (those reading this thread) could learn and make use of.

[sterlind]

maybe WPA, which lets you sample stack traces from user-mode all the way through drivers? Or DbgView, which lets you see printk output from kernel mode. Or hell, maybe even Windbg debugging the kernel of a Windows VM over simulated serial (you need a checked build of Windows to get the most bang for your buck there, though.)

the hardest-core thing I've done was step my way through Windows startup into the container subsystem, on a real, physical target machine I had connected to my dev machine over FireWire. I felt like Indiana Jones. (It helped having the source code though!)

[VRay]

I think Sterlind already covered about as much as could fit in a casual comment online.. The basic idea is that you draw a box around your system, then check all the inputs and outputs. If the inputs are good and the outputs are bad, you open the box and draw boxes around what's inside.

Don't be afraid to dig through heap dumps or decode assembly if you have to, although when you hit that point you're probably just going to have to swap out whatever component is misbehaving