[liampulles]

There is a core 20% to Kubernetes which is very nice, mostly being the Deployment and Service management stuff. That along with a very basic GitOps for cluster management (an infra repo for operators using Flux, applying service level yaml from app repos in CI) above a cloud managed Kubernetes cluster, where you still keep your DB and build servers off the cluster, can be quite nice for a small team.

Beyond that, there are massive holes of despair to fall down if a novice team starts to engage with extensive operators (starving the control plane), DB operators (distributed persistence) and build operators (spikey, expensive loads). At least, I know that I've had to dig out of those holes.

I just hope people don't use k8s in the same way many use microservices: as a way to introduce complexity for complexity's sake.

[zbentley]

Spot on. I have a lot of trouble convincing cloud folks that for durable state, you probably don’t want kubernetes. It’s not that e.g. the CSI drivers and operators for clustered databases aren’t top notch—they are; the era of “avoid stateful kube services” is long behind us—it’s that the cloud provider managed services for e.g. blob stores or databases are so much more reliable. The S3s and Auroras of the world are expensive for a reason: no matter how good your kube native database operator is, it still doesn’t assume responsibility for a ton of the failure points that managed services do. And that’s true even at modest scale (e.g. upgrades are just harder when you’re running your own DB) and in cost conscious environments (sure, the Elasticache bill is steep, but the salary and velocity cost of fixing memory-leak-caused kube memcached crashes is steeper).

[portly]

Also cluster migrations are required pretty often in my company. Having state on a cluster means migrating that as well, which is a complex and time consuming operation. Having your state in S3 or external database makes migrations a breeze.