[rdtsc]

> you really can't just debug a single machine in production when you have a cluster

Somehow I ended up debugging, tracing, monitoring and even hotpatching individual machines in the cluster. Yeah the easy problems will show up in the monitoring and logs. The harder ones won't.

[agentgt]

That must have been a pain in the butt :) . And for sure your right there are always exceptions.

I guess I haven't ran into those issues probably because I run JVMs but I suppose if you have native code or an interpreter using native code I can see how it would be helpful to just SSH and figure out what the issue is.

Now that I recall I have actually had to SSH a bunch of times because of Rackspace network interfaces randomly failing so I am a big hypocrite :)

[rdtsc]

It wasn't a huge pain. It was Erlang, so I could do those very easily. But it still had to be done by logging into a few machines and poking around. I can't imagine if that was somehow a bunch of C code combined with a kernel running in the same memory space.