[gecko]

Probably? I'm welcome to be schooled here. 90% of the time, we can roll back instantly, because there were no database changes. 5% of the time, we can roll back with slightly more pain, because the database migrations were reversible. In this case, the database migration was not reversible. If we'd noticed immediately, we could still have just activated snapshots, but we didn't notice until 20 hours later. What do others do in this situation?

[peterwwillis]

Wait. What blew up that it took someone 20 hours to realize? The first thing you take from that is, don't do anything without double-checking your change to make sure it worked.

In terms of rollback, just don't do anything which isn't reversible. Taking chances with your changes is taking chances with your business. If you don't know how to rollback whatever you're doing, ask someone who does (there is always a way to roll back or add redundancy).

[tedunangst]

"The failed API call turns out to be one that’s trivially cached for a very long time, and so is one that Kiln would allow to fail without actually dying."