[shawnps]

To make this process safer I usually recommend:

  1. Put all new code behind feature flags that are off by default in production.
  2. Make rolling back easy.
  3. Have extensive unit and integration tests.

Some of the deployment steps could be automated even further -- maybe the CI server automatically deploys Staging after a successful build.

See the books Accelerate: The Science of Lean Software and DevOps, and The Toyota Way for more.

[akyoan]

> Put all new code behind feature flags

How is that even possible? A comma change in a feature can break things, are you going to put that change behind a feature flag?

"New code" isn't just "new files/functions", so it's not always feasible to keep it behind flags, unless you use a "copy on write" methodology to all code.

[mkdirp]

Parent commenter probably meant putting new features behind a flag. I work for a major feature management company and we heavily use our own platform. Yet we don't put "all code" behind feature flags but we do with features. It's nearly impossible to put "all code" behind feature flags.

[shawnps]

Right, basically this. If you're shipping something new that could affect production, put it behind a feature flag so the code paths that are already live are unaffected. Continuously ship small changesets so that it's easy to roll back if necessary.

[dpark]

> If you're shipping something new that could affect production…

Which, again, is everything. I’m all for feature flags, but they cover very specific cases. There are many changes that feature flags cannot cover. The addition of the feature flags itself can introduce bugs. They are a great feature but only one small piece of protecting production.