[pavel_lishin]

> You don’t need elaborate Git release rituals. Ceremony such as tagging releases gets to feel like a waste of time once you are releasing many times per day.

What happens when you ship bugged code and need to roll back?

[mcfunley]

The general advice is to roll forward, i.e. remediate the broken thing and deploy. Ideally, the broken deploy was ramping up a config flag and the fix is to turn it back off. (But that's another post.)

Rolling back mechanically is sometimes safe but not always safe, because you have data.

[charleslmunger]

In a large system, every change that comes with stored data needs to have a plan for rolling back - usually it's as simple as "ignore but save" for a new column in the DB, or "write to both, read from old" for a backend migration.

[scott_karana]

`git revert` is perfectly suited for this, since it makes a new "inverse" commit of the changes.

Your production deploys continue monotonically into the future as usual, and you now have version-controlled documentation of the rollback, instead of needing to maintain a separate mapping of production <-> code state.

[byroot]

Having an actual rollback mechanism that brings you back to a known working version is usually way faster than pushing a new version, since it needs to go through CI / Build etc again.

At my company rolling back is a matter of a minute or 2, while pushing a revert is more in the 5 to 10 minutes. Your mileage can definitely vary though.

However it requires all the shipped changes to be rollback compatible, which isn't that hard with a bit of experience.

[pavel_lishin]

Doesn't this assume you only ever deploy one commit at a time?

[scott_karana]

Cherry picks and merges are single commits... and fit well for a lot of dev->staging->release branch workflows.

[flukus]

It's always felt like a waste of time time me, even with SVN. You always know what the released version is and can branch from there if necessary.

[lolsal]

In larger organizations it is very easy to not have any idea what version is currently deployed. I worked for a company that cut SaaS releases every two weeks, but for various reasons the deployed version might be 1-4 months behind current repo HEAD in engineering. It got even more cumbersome deploying different versions for US vs EU and GOV environments.

It's definitely not ideal, but I'd bet it happens more often than we'd like to admit as professionals.

Another thought - at that same company there were I think 5 or 6 different maintained enterprise mobile apps. Jumping between repos and dealing with the various app stores/approval processes it was easy to get lost regarding what was in production.

[flukus]

In everything I've worked on where we cared about the version it has been baked into the binary in an about dialog or info page somewhere, so they only way to not be able to tell which version you were using was if you didn't have access to the app.

This of course relies on having the automated builds set the version numbers properly, it breaks down if it's a manual step, which I'm all too familiar with.

[richardwhiuk]

Just use the commit id, instead of tagging it explicitly

[pavel_lishin]

Doesn't that assume that you only ever have one commit per release? Otherwise, you have to keep a track of "release commits".

[byroot]

It's better to have an activity log of your deployments, that way by querying what commit is currently in production you can compare it with the log and know what actually commit range was deployed.

There is many tools to do that, e.g.

  - Shopify's Shipit: https://github.com/shopify/shipit-engine (author here)
  - Zendesk's Samson: https://github.com/zendesk/samson
  -  Netflix's Spinnaker: http://techblog.netflix.com/2015/11/global-continuous-delivery-with.html
  - Yahoo's Screwdriver: https://yahooeng.tumblr.com/post/155765242061/open-sourcing-screwdriver-yahoos-continuous
  - Instagram's Sauron: https://engineering.instagram.com/continuous-deployment-at-instagram-1e18548f01d1#.tv6zycskx

And the list goes on.