[sofixa]

If Docker Swarm is on the table (it's practically abandonware), I'd like to throw in Nomad by HashiCorp. It's also fairly lightweight, very flexible (can run various types of containers but also exec any executable), decent ecosystem (supports CNI and CSI). It can scale from single node control plane + workloads to tens of thousands of nodes, and the workers can also be geographically spread out (different regions/zones/dcs).

A few years back I wrote about it, and most of the core principles of the article are still valid:

https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...

Disclaimer: I work at HashiCorp, but I've had that opinion since before joining and in fact it's among the reasons I joined

[mdaniel]

I had a distaste for Nomad before the rug pull[1], but hopefully no serious person would voluntarily choose Nomad after that stunt. Doubly so that going with the Nomad ecosystem mandates self-hosting as one cannot $(aws elastic-nomad create-cluster). I was similarly surprised that HashiCorp didn't even offer "you pay for compute but we will administer your control plane" since both the "community" and the "enterprise" links from the front page both say "self-managed"

I will be super curious to see if IBM Cloud actually does ship $(ibm-cloud-cli create-nomad)

1: I also didn't realize they rug pulled Consul, too; that's just cruel https://github.com/hashicorp/consul/blob/v1.20.5/LICENSE

[sofixa]

Agree to disagree on if a license change (to one that impacts nobody but potential resellers) constitutes a rug pull.

BSL and similar are the second best thing (after open source licenses), and still drastically better than proprietary/closed source. If it allows someone to pay the salaries of the people developing it, I'm fine with that. And not only because my own salary is one of them - I had the same view for MongoDB, Elastic, Sourcegraph, etc. In the era where massive behemoths can just ship your software as a service for free, companies need to protect themselves as much as they can. Do you ever wonder why there are very few profitable open source companies? Most people would know Red Hat and that's it.

[chuckadams]

Oddly, I'm moving all my stuff to k8s because of how convenient and easy everything is to admin with tooling like k9s. But I've only ever worked with toy clusters with nodes in the single digits, and I can see what a nightmare debugging something like networking or storage at scale can be.

Also, helm gave me a loathing for yaml I never knew I had in me. I avoid some of helm's more vulgar gyrations by using helmfile, but I'm also giving yoke a look.

[mdaniel]

> Also, helm gave me a loathing for yaml I never knew I had in me

If it's the whitespace that jams you up, I wanted to point out that since YAML is a superset of JSON, there's nothing stopping you from using text/template to cook JSON manifests. There is, of course, a toJson function available in helm <https://helm.sh/docs/chart_template_guide/function_list/#typ...>

But I'm conceptually with you that the jokers who decided to use a text templating language for a structured output were gravely misguided and now we all suffer

[ofrzeta]

There is also JSonnet and the appropriate tooling with tanka.dev