[AstralStorm]

If your process cannot revert a breaking change with a figurative snap of fingers, the process is broken. It is like driving a car with no brakes.

About the only time where I've seen this fail is if it was a publicised feature launch that was way premature.

[jpatokal]

And when it's the build server itself that's broken? No process can account for every possible edge case, and you need to have flexibility to handle the unexpected.

[vonmoltke]

At some point problems reach the "Call Kevin now, I don't care what time it is" level. Your processes should enable anyone to handle problems that are not at this level.

[IOT_Apprentice]

if the build system is broken, then no new code gets built to roll to production. Your build system has no resiliency in terms of having more than one, or ability to roll back? Do you even have version control implemented along with a mature change control process, with automation and monitoring in place?