[100k]

I really enjoy reading how organizations have implemented continuous deployment.

One question I have which is not addressed by the article is how to deal with database changes. Every database has difficulty with schema migrations to one degree or another, but MySQL (which IIRC is what GitHub uses) is particularly bad. In my organization, we are VERY careful with any deploy that contains a migration.

(I suppose this is where GitHub's staging environments with real world datasets come in.)

One workaround I've considered is automatically deploying code that doesn't contain a migration (which is the vast majority) and forcing a more manual approach to database migrations, to make sure people are on hand to roll it back if necessary.

[samlambert]

At GitHub we take a fairly manual approach to migrations, basically the migration is done in its own PR ahead of the PR that has the code.

[tonyhb]

That's awesome! Seems like a really solid setup. Bufferapp took a page from your book and have the same deploy setup with hubot... it works out really well for them, too.

I have three questions to ask about DB migrations (which I can guess the answers to but would love to hear directly), if that's okay:

How do you handle a DB migration with a staged rollout (two 2 of N production servers)?

How do you organise timing between a migration deploy and code deploy if one is done before the other?

How do you handle rollbacks?

[technoweenie]

Migrations stay pretty small. Most of them just add a column or index. We also have to make sure tables aren't too hot to migrate. See a previous post for more on that: http://githubengineering.com/using-mysql-performance-schema-...

It's also really important that migrations don't affect the running app code. New columns shouldn't be used yet. Removed columns or tables need to have all references removed first before running the migration. We confirm this with a deprecation helper that sends metrics to Graphite.

That's about all I can answer from the app side :)

[tonyhb]

Sweet, that makes sense. Thanks for answering!

Was assuming that a column rename would duplicate the column during the first migration — so that the old and new codebase would work correctly. I guess the only complication is that you need to keep track of which branches have been successfully rebased/merged to master so you can run the second cleanup migrations.

Thanks again!

[ykumar6]

Runnable.io makes it very easy to run all your Github branches, using containers. Here's is how it works https://www.youtube.com/watch?v=_uwPKIH990E&feature=youtu.be

[tonyhb]

Do you know how well it works with an SOA setup with multiple repos?

[ykumar6]

We run 10 repos in our sandbox