[rvdmei]

Rolling back is usually a bad practice and can get quite challenging if not impossible in distributed environments.

If you can pinpoint a specific commit that is causing the issue. Revert that commit and go through your standard release process.

[mooreds]

Would love to know more about how you roll forward in that case. Or do you canary so thoroughly that bugs never get to prod?

[rvdmei]

Bugs do get to prod, that’s just reality. If you keep changes small and release often you won’t see many big bugs in production. We typically keep master in an “always deployable” state. If a bug gets found in prod, we either fix it or create a new commit that reverts the commit that caused the problem. If it’s a faulty migration, reverting the commit usually won’t work and we have to fix the bug. For QA we use a mix of automated tests and manual tests.