[nickjj]

What about having a git repo that has a Kustomize base which has overlays for different environments?

This way each environment is in its own directory which can have its own patches such as using a private load balancer instead of public for a staging environment or setting whatever environment variables that need to be different.

Then at the Argo CD level, Argo CD running in prod will look for an Application in the prod/ directory and Argo CD running in staging will look for an Application in the staging/ directory.

All in all you end up deploying the same artifact across environments and all of this is managed and exposed by Argo CD. For example you might want AutoSync enabled for staging so developers can auto-deploy their code but prod requires a tech lead or manager to push the sync button to deploy it.

The above works well in practice and isn't too complicated to pull off.

[therealkrancour]

I am a lead on Kargo. That's a great approach you've described and Kargo helps you make that approach work even better. It's not trying to re-solve problems that are already solved well by config management tools or by Argo CD. It provides a higher layer that integrates with those to orchestrate the progression of changes from one environment to another...

So in your example, suppose you changed something in your Kustomize `base/` directory. With _just_ Kustomize and Argo CD, those changes roll out to all environments all at once. With Kargo, you can progress the change and (re-)validate it environment-by-environment.

[nickjj]

> With _just_ Kustomize and Argo CD, those changes roll out to all environments all at once. With Kargo, you can progress the change and (re-)validate it environment-by-environment.

In prod it wouldn't get rolled out until someone manually clicks the sync button for each app in that environment. But yes, in staging it would get AutoSync'd to all of the apps. I get what you're saying though.

It will be curious to see how Kargo works across multiple clusters in the staging vs prod example where both clusters each have their own Argo CD instance running but you want to view and promote an artifact across N clusters.

In any case, I'll give it a look when it's available to see how it can improve my current workflows. Overall I'm really happy I started using Argo CD ~2 years ago.

[therealkrancour]

> In prod it wouldn't get rolled out until someone manually clicks the sync button

That's a great point and if that works for you, awesome! Note though that relies on the state of the prod Application to temporarily contradict the source of truth that is your repo. If you lose that prod Application and re-create it, prod will be in a state you hadn't yet blessed.

[nickjj]

Indeed. That use case hasn't been something I encountered yet since we generally operate at a scale where our apps at the Argo CD level rarely change once they've been created.

But thanks for pointing out the mismatch on the source of truth. Definitely something to consider.

[denysvitali]

We faced that (and are facing it today) with Helm chart updates.

In our current setup it's not possible to rollout a new version of an Helm chart in dev, then in staging and finally in prod (without causing an out of sync state.

The rest is managed via values-$env.yaml, which works perfectly fine

[spacelad101]

This approach definitely works, but its not the most convenient imo. At least in my experience, this always resulted in separate commits to dev/stage/prod as you progress through each part of the promotion process. It works, sure, but depending on your exact situation it can get cumbersome pretty quickly (for example, start adding in multiple prod regions, or maybe you want to canary one prod region first).

With Kargo, it looks like it lets you define your preferred promotion process, and then lets you release each promotion with a single click. I think the part that is the most interesting to me though is that it writes the changes it is making directly to the git repo.