[leesalminen]

I also had a handful of customers like this. We ended up creating an opt-in “beta” channel that we treated as a first-class citizen along with “prod”.

Basically we had 2 production stacks with CI/CD for each. The only shared thing was the AWS ALB. Our customers each have their own subdomain so we used that to point to different EC2 ASGs.

We would continue pushing bug fixes to “prod”. Once a beta period ended, we merged to master and re-deployed master to both prod and beta channels. Rinse and repeat.

It works out pretty well. Occasionally there’s a really important bug fix that we have to cherry pick from prod to beta, but it’s rare enough to not be a huge concern.

Oh, and, charge them for it. I never got any pushback.

[cbono1]

Thanks for sharing!

Was the beta channel that you had in the pipeline for prod? i.e. prod would have no chance of getting ahead of beta because prod was always deployed based on what was last on the beta release?

Hot or bug fixing is an interesting topic because you could end up with diverging histories if you're not careful. Did you try to maintain `master` as the single source of truth or did you make separate release and deploy branches for each environment?

[leesalminen]

We created rc/xxx branches in git. Developers would merge feature branches destined for beta to the rc. The rc gets deployed to the beta environment. Later, we would merge the rc to master and then deploy master to both prod and beta environments.

Prod would be ahead of beta during hot fixes. Though we try and avoid it, it did happen occasionally.

You are correct that it took some extra effort to keep it all clean and manageable.

[jpmoral]

Do the two stacks each have their own database? Have you had customers wanting to move back and forth between channels?