[cyberax]

I don't want microservices!

What I want is a lightweight infrastructure for macro-services. I want something to handle the user and machine-to-machine authentication (and maybe authorization).

I don't WANT the usual K8s virtual network for that, just an easy-to-use module inside the service itself.

You should be able to spin up everything localy in a docker-compose container.

[LaurensBER]

> What I want is a lightweight infrastructure for macro-services. I want something to handle the user and machine-to-machine authentication (and maybe authorization).

> I don't WANT the usual K8s virtual network for that, just an easy-to-use module inside the service itself.

K8s makes sense if you have a dedicated team (or atleast engineer) and if you really need need the advanced stuff (blue/green deployments, scaling, etc). Once it's properly setup it's actually a very pleasant platform.

If you don't need that Docker (or preferable Podman) is indeed the way to go. You can actually go quite far with a VPS or a dedicated server these day. By the time you outgrow the most expensive server you can (reasonable) buy you can probably afford the staff to roll out a "big boy" infrastructure.

[cyberax]

I tried to use K8s several times, and I just can't make it work. It's fine as a deployment platform, but I just can't justify its complexity for local development.

We're using Docker/Podman with docker-compose for local development, and I can spin up our entire stack in seconds locally. I can attach a debugger to any component, or pull it out of the Docker and just run it inside my IDE. I even have an optional local Uptrace installation for OTEL observability testing.

My problem is that our deployment infrastructure is different. So I need to maintain two sets of descriptions of our services. I'd love a solution that would unify them, but so far nothing...

[stackskipton]

I wouldn't use K8s for local development unless you have some system where there is a dev cluster and you can route traffic for particular pod to your local workstation.

Docker Compose for local development is fine. If your K8s setup is crazy complex that you need to test it locally, please stop.

[fragmede]

Tilt? Skaffold? configuration isn't free, but a debugger on a local k8 cluster that's at least somewhat representative of prod is pretty handy once you do.

[cyberax]

I tried Tilt, but it's still too complicated. For example, we have a computer vision model that is a simple Python service. When developing on macOS, it's not possible to use GPUs inside containers, so we need to run it locally on the host.

It's trivial with my current setup, but not really possible with Tilt.

[ghthor]

I know this pain, though we’re running nomad not k8s as our cluster control plane. But local devs are stuck with docker-compose, so 2 different configurations for running locally versus in the production environment.

[rahen]

Unless you need horizontal scalability or clustering, Compose + Terraform is all you need.

With Compose, you get proper n-tier application containerization with immutability. By adding an infrastructure-as-code tool such as Terraform to abstract your IT environment, you can deploy your application on-premises, in the cloud, or at a customer site with a single command.

For clustering needs, there’s Incus, and finally, Kubernetes for very fast scalability (<mn), massive deployments on large clusters, cloud offloading, and microservices.

Almost nobody truly needs the complexity of Kubernetes. The ROI simply isn’t there for the majority of use cases.