[INTPenis]

I felt like I got on the k8s bandwagon relatively late, due to my natural distrust of new things, in this case containers.

So I started setting up k8s clusters on-prem at work in 2019.

It's 3.5 years later and my takeaway is that k8s should only be used after multiple resource planning meetings have established that it is absolutely necessary, or you're scaling up an existing application.

The mistake we made was to first of all take orders from a lead developer who thought that service mesh was the solve all. And secondly to not estimate the load our system would generate, the resources it would require and how to best manage those.

In retrospect we now have 3 clusters (dev, staging, prod) for a service that could be hosted on 2 servers, with a job manager in the cloud.

And IaC is no excuse because you can achieve the same IaC with container hosts and quadlets set to auto-update.

[nucleardog]

Completely agree. Kubernetes makes some problems easier to solve, but it's rare that anyone asks "do we actually need to solve these problems". It's like buying something you don't need because it's on sale.

I've found a happy middle ground to be Hashicorp's nomad. It's a single static go binary you can run so you can define some of your setup in a repeatable way as well as provides things like rolling updates, task monitoring, scheduled jobs, etc. And it's not limited to only running containers, but can run executables directly on the host, VMs with qemu, etc.

If I'm running a single server, I usually get a lot of value out of throwing nomad on it.

[louwrentius]

I bet this is probably true (app would have been fine on a hand full of servers) for most Kubernetes installations.