[westajay]

In other industries, such as petroleum refining, pipelines and nuclear power, their are structured methodologies for determining root causes. Some of these take into account equipment failure, and modelling is often done on equipment life cycles to determine replacement and inspection schedules in oil refineries.

These industries also employ strict management of change processes so that an ad-hoc decision or improvisation doesn't cause an incident.

What puzzles me is why you don't see these kind of practices applied in data center operations.

Are data centers really more complex.. then say.. a nuclear plant?

[randomdata]

What puzzles me is why you don't see these kind of practices applied in data center operations.

Simply, people are not willing to pay the price. All of that extra planning requires more man hours. It also requires people, who come at significant cost, skilled at working in such environments. People want to use their Blackberries for hundreds of dollars up front and tens of dollars each month, not thousands of dollars monthly.

It can be done, but the market has determined that it does not want it to be done. It would rather accept some downtime and other problems in order to access the technology at a lower cost.

[westajay]

When I wrote my comment I wasn't thinking of lots of up-front planning. I was thinking along more simple lines like root cause analysis using a human factors or equipment taxonomy (much more affective then 5-whys).. and simple logging of incidents for later analysis.

I think some of these kinds of processes can be adopted with small investments in training and change.

Also, a lot of these kind of failures seem to stem from changes at the networking layer.. which should be more planned and tested given their place in the stack (we're not talking about crazy app behaviour).

[tallanvor]

You'd better believe that companies like Amazon, RIM, and all the others have very strict change management procedures. Changes to critical systems are generally planned weeks - sometimes months - in advance, and even simple changes often require multiple levels of approval. Even then, sometimes something gets missed, a component acts slightly differently than expected, or somebody makes a mistake.

All of the mistakes the article discussed were network related, and yes, when you're dealing with networks like Amazon's or RIM's, where there are hundreds of thousands or even millions of devices connected, they are definitely more complicated than a nuclear plant.

[feralchimp]

Arguably so, in that app-level data has a richer set of failure modes than even highly-complex physical materials and flows, but I don't think that detracts at all from your main point.

The linked article used the word "unknowable" to describe the failure modes in a complex datacenter. They're definitely "unknown," but it's way too early to just give up on figuring many more of them out.

If a CTO says "it'll cost $50B to reach less than 5% unknown failure modes," that's one thing. But if a CTO says "yeah, the faulre modes of our critical infrastructure are unknowable" they better start looking for a new gig.

[shaggyfrog]

>In other industries, such as petroleum refining, pipelines and nuclear power, their are structured methodologies for determining root causes.

I don't see why they would need a domain-specific methodology. Toyota gave us the 5 Whys, which applies nicely to just about any kind problem.

http://en.wikipedia.org/wiki/5_Whys

[joezydeco]

My three year-old daughter uses the same technique, she has no idea who Toyota is. Thankfully the root cause usually ends up being "BECAUSE I SAID SO".

[fleitz]

Yes, data centers are more complex than Nuclear power plants. Nuclear power plants generally operate in well-known ways and with few unknown interactions.

eg. the work load on a nuclear power plant system does not spike around Christmas

the work load on a nuclear power plant system does not change when there is a video spreading virally

the nuclear power plant control system does not have to support a diverse eco-system of clients, some of them running hostile code.

the nuclear power plant control system does not need to be upgraded very often

Modern data centres and the infrastructure that runs on them operate in an environment of constant flux, this makes them vastly more complicated than a nuclear power plant system.

Nuclear power plants have a vast number of components and a few very catastrophic failure modes but in essence they're very simple. The components don't change (save for the well known ways in which metal degrades under neutron bombardment), and lots of money has been invested to know exactly how these components act under various stresses. The engineers who operate them operate the same system for years and years becoming experts on how their very well defined system operates and know intimately its exact failure modes, and have a SOP manual and various checklists to quickly pinpoint and eliminate problems.

Nuclear power plants are also developed and operated under a very clear singly focused mandate: Prevent at any cost the Nuclear Power plant from entering a catastrophic failure mode.