[cedws]

K8S is overrated, it's actually pretty terrible but everyone has been convinced it's the solution to all of their problems because it's slightly better than what we had 15 years ago (Ansible/Puppet/Bash/immutable deployments) at 10x the complexity. There are so many weird edge cases just waiting to completely ruin your day. Like subPath mounts. If you use subPath then changes to a ConfigMap don't get reflected into the container. The container doesn't get restarted either of course, so you have config drift built in, unless you install one of those weird hacky controllers that restarts pods for you.

[Thaxll]

It's not slightly better it's way better than Ansible/Puppet/Bash/immutable deployments, because everything follow the same paterm and is standard.

You get observability pretty much for free, solution from 15 years ago were crap, remember Nagios and the like?

Old solutions would put trash all over the disk in /etc/. How many time did we have to ssh to fix / repair stuff?

All the health check / load balancer is also much better handled on Kubernetes.

[romaaeterna]

I wouldn't throw away k8s just for subPath weirdness, but I hear your general point about complexity. But if you are throwing away Ansible and Puppet, what is your solution? Also I'm not entirely sure what you are getting at with bash (what does shell scripting have to do with it?) and immutable deployments.

[cedws]

That's only one example of K8s weirdness that can wake you up at 3am. How: change is rolled out during business hours that changes service config inside ConfigMap. Pod doesn't get notified or reload this change. Pod crashes at night, loads the new (bad/invalid) config, takes down production. To add insult to injury, the engineers spend hours debugging the issue because it's completely unintuitive that CM changes are not reflected ONLY when using subPath.

[yabones]

That's totally valid. I understand the desire of k8s maintainers to prevent "cascading changes" from happening, but this one is a very reasonable feature they seem to not support. There's a pretty common hack to make things restart on a config change by adding a pod annotation with the configmap hash:

      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

But I agree that it shouldn't be needed. There should be builtin and sensible ways to notify of changes and react.

[stackskipton]

This is argument for 12Factor and Env Vars for Config.

Also, Kustomize can help with some of this since it will rotate the name of ConfigMaps so when any change happens, new ConfigMap, new Deployment.

[8n4vidtmkvmk]

That's how I do it, with kustomize. Definitely confused me before I learned that, but hasn't been an issue for years. And if you don't use kustomize, you just do... What was it kubectl rollout? Add that to the end you deploy script and you're good.

[romaaeterna]

I told you that I hear you on K8s complexity. But since you throw out Ansible/Puppet/etc., what technology are you advocating?