| The DX of k8s is deceptively simple. You might be forgiven for believing that you have got your own private Heroku in your own backyard which you totally own and control. But - Oh boy! The complexity over complexity of moving parts that themselves are a moving target sometimes. First, there's PKI which you should know all about certificates, signing, expiring, issuing, reissuing or no part of the cluster talks to the other. If you think you can get away with that, the post above had two outages both related to certs. Next is the etcd - you should know how to configure that. A totally different separate product in its own right, a distributed key store that has whole memory of the system. Like what's where and such. Then you have whole DNS running. That again is a whole separate product in its own right whose administration you must master or else. And then comes Networking, the CNI plugin and their internals and if you think you can skip that part, either you have to pay likes of weaveworks (defunct) or Cillium etc or be ready for an incident. And yet I have not talked about ingress controllers, cloud controllers, their configurations and other issues. To top that all, you need to manage all that configuration and package it so now you need Helm and Flux - templated (Helm uses Go templates, if not worst out there) and layered (kustomised) YAML upon YAMl, thousands of lines and that's not just all, hold on! All that configuration language is constantly a moving target from Helm Charts to k8s manifests. Sometimes totally incompatible (like flux to flux2 was almost not so portable) and such so upgrades are going to be so much painful, you just can't imagine even if you're on a managed k8s platform. I say this from my own experience of setting up k8s self managed from scratch across different clouds. I have my scars. |