[catherinecodes]

This is definitely a hard problem.

One technique is to never upgrade clusters. Instead, create a new cluster, apply manifests, then point your DNS or load balancers to the new one.

That technique won't work with every kind of architecture, but it works with those that are designed with the "immutable infrastructure" approach in mind.

There's a good comment in this thread about not having your essential services like vault inside of kubernetes.

[jauntywundrkind]

This indeed seems like The Way but I have no idea how it works when storage is involved. How do Rook or any other storage providers deal with this?

If Kubernetes is only for stateless services, well, that's much less useful for the org to invest in.

[unethical_ban]

Any state that a container uses, such as databases or static assets, should be mapped to something outside k8s, no? I thought container orchestration was only for app later

[prng2021]

In the early days that was true. K8s has had many options for stateful containers for a while though.

https://kubernetes.io/docs/concepts/storage/

[prmoustache]

we are talking in a context where you would spin up a new cluster whenever you want to upgrade kubernetes version.

In that case you don't want to migrate application/user data so you are kind of forced to keep DBs and filesystems outside.