[sph]

That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes.

I can do both, but for a bootstrapped solo business, Kubernetes is overkill and overengineered. What I would really love is a multi-node podman infrastructure, where I can scale out without having to deal with k8s and its infernal circus of YAML, Helm, etcd, kustomize, certificate rotation, etc.

Recently I had to set up a zero-downtime system for my app, I spent a week seriously considering a move to k3s, but the entire kubernetes ecosystem of churn frustrated me so much I simply wrote a custom script based on Caddy, regular container health checks and container cloning. Easier to understand, 20 lines of code and I don't have to sell my soul to the k8s devil just yet.

Sadly, I don't think a startup can help make this better. I want a bonafide FOSS solution to this problem, not another tool to get handcuffed to. I seem to remember Red Hat where working on a prototype of a systemd-podman orchestration system to make it easy to deploy a single systemd unit into multiple hosts, but I am unable to remember what is it called any more.

---

Also, I seem to be an outlier, judging from the rest of the comments, by running on dedicated servers. These days everybody is using one of the clouds and terribly afraid of managing servers. I think it's going to be hard to make DevOps better when everyone is in the "loving" Azure/AWS/GCP embrace: you're basically positioning as their competitor, as the cloud vendor itself is always trying to upsell its customers and reduce friction to as close to zero as possible.

[denvrede]

> Recently I had to set up a zero-downtime system for my app, I spent a week seriously considering a move to k3s, but the entire kubernetes ecosystem of churn frustrated me so much I simply wrote a custom script based on Caddy, regular container health checks and container cloning. Easier to understand, 20 lines of code and I don't have to sell my soul to the k8s devil just yet.

I'd say it depends where you're coming from. For me, setting up a Kubernetes cluster (no matter which flavor) with external-dns and cert-manager will most likely take 30m-1h and that is the basic stuff that you need for running an app with the topics you mentioned. To navigate through k8s just use k9s and you're golden.

I never get where all the "k8s is the devil" comments come from. There is nothing really complex about it. It's a well defined API with some controllers, that's it. As soon as I need to have more than one server running my workloads I would always default to k8s.

[sofixa]

> I never get where all the "k8s is the devil" comments come from. There is nothing really complex about it. It's a well defined API with some controllers, that's it.

And Linux is just a kernel with some tools, which are all well defined, that's it!. But if you need to debug a complex interaction "that's it" and "it's well defined" isn't enough.

Kubernetes is quite complex, with a lot of interactions between different components. Upgrades are a pain because all those interactions need to be verified to be compatible with one another, and the versioned APIs, as cool as a concept they are on paper, mean that there's constantly moving targets that need constant supervision. You can't just jump a version, you need to check all your admission controllers, CNI and CSI drivers, Ingress controller, cert-manager and all other things are compatible with the new version and with each other. This is not trivial at any scale, which is why many orgs adapt the approach of just deploying a new cluster, redeploying everything to it and switching over, which is indicative of exactly how much of a pain it is.

Even Google themselves that created it admit it's complex and have 3 managed services with different levels of abstraction to make it less complicated to use and maintain.

[sph]

k8s is fine. It's the ecosystem around it that I dread.

I understand how it works, it makes sense, but then you're faced with Helm, kustomize, jsonnet and a lot of bullshit just to have minimal and reproducible templating around YAML. Or maybe you should use Ansible to set it up. Maybe instead try ArgoCD. Everybody and their dog is an AWS or GCP evangelists and keep trying to discourage you from running it outside of the blessed clouds. If anything breaks you're told you're stupid and that's what you get, and I should've paid someone else to manage it.

It feels like everybody is selling you something to just manage the complexity they have created. This is what keeps me away. It's insane.

[supriyo-biswas]

There’s also Docker swarm for containerized workloads.

Personally I like VMs, and I’ve been toying around with the idea of having a LXD cluster on dedicated servers where each LXD container hosts some of my workloads.

[rcarmo]

This works. I had that setup ages ago, and you can nest Docker inside LXD. I’m using Proxmox now to manage the VMs, LXCs and storage underneath Portainer, and it is surprisingly trivial to snapshot and move an entire Portainer node around in Proxmox. One click to back up to shared storage, another to restore in another physical node.

[sph]

I set up Docker Swarm at my previous company, but it was a dead and stale project 5 years ago when I last used it. I honestly cannot recommend it in 2024.

[Hortinstein]

I don't think there has been a ton of iteration on it, but did you run into any specific problems or bugs or is this lack of recommendation based off caution against adopting something that is not being iterated on? Just asking because while I haven't used it in years, it's been my go-to for small projects in the past, it seemed to do what it advertised very well. I hope that someone picks up the swarm torch, I really liked the abstractions and workflow it enabled. K8s was always too heavy for me and introduced too much complexity I was uncomfortable with

[niteshade]

Not GP, but I recently attempted to migrate a single-node Docker Compose setup to Docker Swarm and ran into the following issues:

- No ability to use secrets as environment variables, and no plans to change this

- Cannot use `network_mode` to specify a service to use for network connections a la Docker Compose

There were a few other minor issues which resulted in ditching Docker Swarm completely and moving to a Nomad + Consul stack instead.

[ancieque]

While the way secrets work in Swarm seems weird when compared to Kubernetes, this is usually pretty easily solved by a quick overriding entrypoint in the docker stack file that does essentially this:

    export SOME_VAR=$(cat /run/secrets/some_secret)
    exec /original/entrypoint.sh

Can you explain the second one? I don't get the usecase.

[pimsn3000]

I just set up my first docker swarm cluster. It's not in production yet, just a stage environment, but it is working very well so far, and I like it very much. From my experience so far I can very much recommend it, and I hope it will get more attention again. Because it does fill the gap which is described in the original post.

[sofixa]

> That these days there are only two serious choices: deploy on a single-node Docker/podman machine, or Kubernetes.

I disagree - there's cloud PaaSes like Fargate or Cloud Run, or when self-managing, a third option which is much less known but quite easier while also being more flexible - HashiCorp Nomad. Disclaimer time: I work at HashiCorp, but I've had this opinion for years before joining - https://atodorov.me/2021/02/27/why-you-should-take-a-look-at... (it's out of date, but the principles still apply, just to a different extent, Nomad having gotten easier). All opinions are my own etc.

Not FOSS anymore, but free and source available, composable, and does a big portion of what Kubernetes does at a fraction of the complexity. I ran it in production for a few years and everything was a breeze, unlike the Kubernetes clusters I was maintaining at the same time. You get all the "basics" of HA, failover, health checks, easy ingress, advanced deployment types, basic secrets storage, etc. without having to write thousands of lines of YAML, and with simple upgrades and close to no maintenance.

> Also, I seem to be an outlier, judging from the rest of the comments, by running on dedicated servers. These days everybody is using one of the clouds and terribly afraid of managing servers. I think it's going to be hard to make DevOps better when everyone is in the "loving" Azure/AWS/GCP embrace: you're basically positioning as their competitor, as the cloud vendor itself is always trying to upsell its customers and reduce friction to as close to zero as possible.

Because running on dedicated servers means you spend time on managing them which could be more productively spent working on whatever your product is. Don't get me wrong, I love that stuff, but it's like when I started my blog - I spent weeks on CI/CD, a nice theme, a fancy static site generator, all sorts of optimisations, choosing a good hosting provider etc. etc. etc.... instead of actually doing what I was supposed to, writing content. When you manage your own servers your costs might even be higher compared to a F/C/P aaS from a cloud provider, especially with free trial year/free tier/startup credits. As long as you keep an eye on not locking yourself in, you could easily migrate most products to self-managed dedicated servers if performance/costs require it.

[sph]

This is silly but I will not use any Hashicorp product ever again, for two reasons: one is Terraform (it is a bad product with a bad DSL), the second is that I bought one service from them that advertised a clear 30-day money back guarantee, found a bug in one hour of running, a multiple-year old Github issue they had been ignoring, and to make it worse, they completely ghosted me when I went to ask for a refund. Both on Twitter and via email.

So, I don't really care to try out Nomad for my business, honestly.

[sofixa]

Nah, it's not silly - you had bad experiences with a company, it's normal not to want to interact with them further. I'm personally not a fan of Microsoft because I spent my formative tech years installing Windows on family friends' PCs and had so much inconsistencies and issues that I have a very negative association with Windows and Microsoft.

I disagree with the DSL part, but it's in big part subjective so people have all sorts of different experiences and opinions.

[icedchai]

Terraform may seem bad, but cloud-specific solutions like Cloud Formation are even worse...

[datascienced]

Lambda? Digital ocean app service? Azure containers?

I guess it is proprietary tech though which might be off putting.