I really liked Netflix's approach to testing backup scenarios by creating the Chaos Monkey[1] (Jeff Atwood had good insight as well[2].) The idea of creating a mechanism to test fail over scenarios is something that I wouldn't have thought of prior to the transparency that companies like Grasshopper have shown. So hat tip to them for opening up on a large failure; It makes the rest of the dev community smarter/better because of their honesty.