[jspaur]

We do have a failure plan (and ironically enough were actively working on improving storage resiliency, this just hit us before we were able to deploy it)

That being said, there unfortunately is always a level 'sitting on hands' in events like this. If you're in dev you rely on your operations people (and vicea versa). You put your best people on it and continue to go on the best you can.

All that being said, a fired rill just for the sake of doing something may cause more harm than good. If a rollover plan takes 2 hours (switch DNS, migrate data, test, stabilize, etc.) a) often times that'll take longer than the original issue requires to be resolved and b) you'll run into new issues in your new location (maybe environmental, maybe not)