[mattmanser]

Then you've done it wrong. Very, very wrong. You should be able to replicate your live environment locally and not being able to is a sign of a very broken development process.

I just cannot emphasise this enough. One of my skills is debugging, everywhere I've worked I've been one of the top debuggers if not the top debugger.

Never, ever debug on live. If you can't recreate the problem locally, you don't understand the problem yet. Keep trying to replicate it until you do. I'm talking days of trying, not 10 bloody minutes.

The key to all debugging, the very fundamental skill, is consistent replication. If you can replicate a problem consistently, you can fix it.

There's a class of problems where this isn't necessarily true, especially for desktop apps, but web development problems and javascript, you can almost always replicate locally.

[vidarh]

Sometimes your problem turns out to be a problem with production hardware or wiring, or depending on interaction with external partners who may not be able to replicate their interactions accurately for your dev/test environment, or otherwise impossible for you to replicate in dev/testing without already having figured out what the problem is.

While I agree with your overall sentiment, in any kind of remotely complex environment you sooner or later come across problems that you have no sensible way of reproducing in dev, and which will require you to debug on a production environment without it saying anything about the quality of your development process.

[mattmanser]

Bollocks, even in complex environments you should be able to test it outside. You can hook dev machines up to external partners, so there's absolutely no good reason to be using live environments for those problems.

I have seen production hardware or wiring be the problem once in my entire career. And I used to work for a company that ran and supported enterprise software on hundreds of different client-owned servers.

Most people these days will be running a few web servers at most.

EDIT: Worse still, we're talking about debugging javascript here, which makes what you're saying even more preposterous. Wiring, indeed.

[SeanDav]

> Never, ever debug on live. If you can't recreate the problem locally, you don't understand the problem yet.

I am guessing you have not been exposed to highly complex live environments. What you say is largely true but you state it as an absolute fact, which is a bit naive. Live environments tend to be far more complex than test or local environments and sometimes that complexity is what creates the bug. Sometimes the only solution is to debug live, but of course that is a last resort.

[mattmanser]

For 99% of programmers who aren't ever going to deal with the really tricky problems, it should be gospel.

I'm not naive, I did actually say "almost always" and "There's a class of problems where this isn't necessarily true", but perhaps you can't read?

[teh_klev]

> Then you've done it wrong. Very, very wrong. You should be able to replicate your live environment locally and not being able to is a sign of a very broken development process.

I feel this is statement is a tad hyperbolic.

One of my key skill sets is debugging, I'm pretty good at it. One of the things we can't do is completely replicate a production environment because we're a web hoster. Sure I can run and test stuff against our OS "gold images" but when you've added third party client code running on 300-400 sites on a single server that environment is not reproducible without a herculean effort. We also don't have the luxury of time to "Keep trying to replicate it until you do. I'm talking days of trying".

Four or five times a year I need to dig out ADPlus, point it at a customer's worker process and tell it to snapshot and dump when certain conditions/triggers occur. I then whip said dump over to my WinDbg+SOS environment to find out what the hell was happening in the process when it went bad.

[tildedave]

I think there's a lot true in what you've written. However the root cause fix applied in a development environment is not always the fastest way to stop a user facing problem. Things get weird on live servers. I'm not suggesting altering code willy-nilly but a solid knowledge of production (logging, service status, etc) can lead to more effective debugging scenarios and faster time to resolution when things do go bad.