[avianlyric]

Personally I like to treat the Git in GitOps as an implementation detail. The underlying principle of GitOps is that your entire environments setup and config is written in code, and version controlled. So you can in theory pick any version of your entire env, throw it at blank slate, and reliably get the environment specified by that git hash.

Then there’s the whole constant reconciliation of your version controlled env specification, and the actual env, and how you automatically resolve differences. With the most important principle being that the version controlled code/config is absolute truth and something needs to figure out how to bend the world to match.

But importantly in all of this, Git isn’t that important. Version control is important, infrastructure as code is important, but Git isn’t. Arguably Git isn’t a great tool for GitOps due to issues like the ones you mention. But the huge ecosystem around Git makes the pain worth it.

I would argue the “correct” solution to your problem is a tool that automatically creates the correct cherrypicks and reverts for you based on a request to rollback application X.

Treat git as a dumb version control system, and broadly ignore “good practice”, because at lot of those good practices are designed for software development, not infrastructure development. We need to develop new working practices, built on top of Git fundamental components, rather than trying to rationalise existing working practices against the new problems that appear in GitOps.

[hdjjhhvvhga]

> So you can in theory pick any version of your entire env, throw it at blank slate, and reliably get the environment specified by that git hash.

The trap here is this only works for stateless infrastructure. If you do it with stateful resources, you'll lose all data. Your gitops tool will happily recreate EC2 instances, S3 buckets and RDS instances, all empty/initialized to whatever you defined.

[anomaloustho]

There are 2 different thoughts here I think. If using GitOps in Kubernetes, then application and set up (Pods) aren’t associated with Nodes (EC2). And both can be torn down and rebuilt without state issues. When state is required, then PVCs and Stateful Sets come into play.

For managed services like S3 and RDS, there are other GitOps tools like Crossplane.io which you can use for similar GitOps management. But the paradigm shift might also be that you add GitOps config to perform regular backups, and also add config to ensure that if it is being recreated, it restores from a backup.

[benterix]

> When state is required, then PVCs and Stateful Sets come into play.

And that's the problematic part. With GitOps, you manage the infrastructure, so in the case of PVCs these are PVC manifests - you need to manage data separately. But even if you exclude the data, even some PVC manifest changes like size change can be tricky. Also some properties are immutable (like its storage class, access modes etc.) so that you cannot modify a PVC without recreating it.

You can decide you want to avoid the problem completely and store your data outside of Kubernetes. Now you have two problems...

[wg0]

Thank you for the insights. Last part specifically. I also feel something is surely amiss.