[randomdata]

What puzzles me is why you don't see these kind of practices applied in data center operations.

Simply, people are not willing to pay the price. All of that extra planning requires more man hours. It also requires people, who come at significant cost, skilled at working in such environments. People want to use their Blackberries for hundreds of dollars up front and tens of dollars each month, not thousands of dollars monthly.

It can be done, but the market has determined that it does not want it to be done. It would rather accept some downtime and other problems in order to access the technology at a lower cost.

[westajay]

When I wrote my comment I wasn't thinking of lots of up-front planning. I was thinking along more simple lines like root cause analysis using a human factors or equipment taxonomy (much more affective then 5-whys).. and simple logging of incidents for later analysis.

I think some of these kinds of processes can be adopted with small investments in training and change.

Also, a lot of these kind of failures seem to stem from changes at the networking layer.. which should be more planned and tested given their place in the stack (we're not talking about crazy app behaviour).