[kls]

The sad part is the authors example is easily avoidable and does not cost more to avoid. The fact that a developer was making manual changes in a production environment points to bad process and lack of knowledge of a better way. In every environment I have been in since 2000 production is locked down it is modified by scripts that are tested in several environments before the script even gets to production. With virtualization and continuous integration the cost of doing it correctly is marginal and the savings in avoiding crisis and recovery is substantial. There in lies the core issue, it seems more and more developers are loosing focus of the process of managing change across environments and are resorting to manual processes that are not reproducibly. Simple point is, if you are making manual changes in any environment other than a local dev environment you are doing it wrong.

Source control should be your single source of change, and check-ins should cause a chain of events via continuous integration and issue tracking. A checkin should cause a build to kick off if it is successful it should deploy to an integration environment where tests should smoke test it. If it smoke tests it continuous integration should up merge to a test branch, deploy to test and then update the ticket system to update all associated tickets to ready for test. When the tester verify the tickets automation should again up merge to release and automation should release to production on a release schedule. This is a solved problem.

[jack9]

> The sad part is the authors example is easily avoidable and does not cost more to avoid.

Except for all the costs of maintenance. It always costs more, generally in the way of time.

> With virtualization and continuous integration the cost of doing it correctly is marginal

Until you have to debug while the integration fails.

> This is a solved problem.

Even with unlimited funds, you get a cost of time and communication. There are tickets and tickets and a queue and a release schedule. The optimal path is not as simplistic as you would fantasize.

[d2p]

> The fact that a developer was making manual changes in a production environment points to bad process and lack of knowledge of a better way.

I'm guessing this is relating to my SQL mishap in the opening lines? You're making assumptions that I was doing something manually in production here, and not that I was executing a script that was buggy or against the wrong machine.

Sure, we could build safeguards against this, but it wasn't something we really considered would happen. Needless to say, we've learned from that one!

[x5n1]

Hot patching is a standard practice in production. It's sometimes necessary.