[mcfunley]

The general advice is to roll forward, i.e. remediate the broken thing and deploy. Ideally, the broken deploy was ramping up a config flag and the fix is to turn it back off. (But that's another post.)

Rolling back mechanically is sometimes safe but not always safe, because you have data.

[charleslmunger]

In a large system, every change that comes with stored data needs to have a plan for rolling back - usually it's as simple as "ignore but save" for a new column in the DB, or "write to both, read from old" for a backend migration.