[jpp]

We have a similar setup — containers managed by Fargate — and solved the issue of migrations by:

1) having our app containers include the DB migration logic 2) on container startup, “check and run migrations” before app startup, 3) the trick: acquire a lock in postgres as part of step 2, so that only one node at a time can run migrations.

Migrations are run inside of a begin/commit, so with the lock we have reliable guarantees that 1) exactly one container at a time can try to run the migration; 2) the migration either completes or fails.

This setup has the benefit too that, if the migration fails for whatever reason, no new app containers will start in production. That is: we can basically trigger a production deploy and if it fails, the deploy halts and the previous app version remains up and serving traffic.

There are better ways of handling this, but for us at our scale, this has been both very simple and very reliable, which makes me happy. :)

[nthj]

As I understand it, the concurrency-safe lock is built into Rails for a while. Step 3 shouldn’t be necessary.

https://github.com/rails/rails/pull/22122

[jpp]

Ah, very nice! Our setup is on python, and because of some other dev tooling we have, it’s very easy for us to have a simple decorator function that grabs the advisory lock. We can apply that decorator to any python function, including the one that triggers our migrations. If on rails, seems like using the above link would be better.