[peterwwillis]

Wait. What blew up that it took someone 20 hours to realize? The first thing you take from that is, don't do anything without double-checking your change to make sure it worked.

In terms of rollback, just don't do anything which isn't reversible. Taking chances with your changes is taking chances with your business. If you don't know how to rollback whatever you're doing, ask someone who does (there is always a way to roll back or add redundancy).

[tedunangst]

"The failed API call turns out to be one that’s trivially cached for a very long time, and so is one that Kiln would allow to fail without actually dying."