[himata4113]

I don't know... running a startup sized kubernetes is relatively easy and pain free these days (k3s). Especially when it comes to scaling up.

CNPG is an absolute monster (in a good way). cert-manager is easier than the docker alternative, calico has never failed me (except in bgp mode which has some footguns like not being able to come back from a dead state since it has a chicken and an egg problem unless you point it to the external load balancer which I would have known if I read the documentation). trafeik is all you need. talosos largely mitigates the bare metal problems and comes pre-hardened and pre-optimized.

I solo most of my development projects and have used k3s for all of them. The only complaint is that cert-manager by default will fail silently and your certificates will expire. I largely mitigated this by having proper visibility setup via grafana and automated alerts (warns if certificates are about to expire) which should have been done by me anyway.

Two years ago I'd agree, today with LLMs everything I have runs talos with fully automated updates and I haven't had to be on-call for almost a year.

[makeitdouble]

I think parent would wish for something close to what heroku represented (what would it be ?)

K8s is easier at smaller scales (I understand k3s as a packaged version ?), but you still need one or two people in your team to properly understands all of the concepts and inner workings of k8s, and be able to neck deep into if/when shit hits the fan.

For a small team that's a lot of commitment for something that is usually not their bread and butter and wish they could build once and only slightly tweak every year or so.

[englishspot]

even with just k3s and a few plugins/operators, it still takes someone dedicated to babying it. I've been running a k3s cluster at home for years and I dread upgrading all the things running on it, and all the things needed to keep it running.

and more to that last point, we haven't talked about maintaining the actual nodes themselves yet.

[himata4113]

if you don't use alpha or beta annotations you rarely have to worry about updates, kubernetes has a very strong "do not break it" policy on non beta/alpha annotations.

[mikeocool]

Not using beta ingress was probably a non-starter for a lot of people, since it was the only option for 4 years.

Then there was an upgrade process that required a fair amount coordination between when you changed your manifests, when you upgraded your cluster and when you upgraded your ingress controller.

PodSecurityPolicies also gained a lot of traction and didn’t really have an alternative before it was deprecated.

Also, custom operators don’t all subscribe to the don’t break non-beta resources in the same way core does.

[KronisLV]

> cert-manager is easier than the docker alternative

  MDomain blog.kronis.dev

I'm not saying that cert-manager isn't nice, but with regular Docker/Compose/Swarm setups you can just run a web server/load balancer on whatever ports you want. With mod_md the above is all I really need in a regular .conf file to provision LetsEncrypt certs for my blog (very similar with something like Caddy too). And it's the same in Docker as it is when running the web server directly, I think that's why starting with Docker is really nice, because it has fewer custom abstractions and sometimes regular software does things elegantly already.

[mikeocool]

I’d take a cronjob running cert-bot and some monitoring to ensure a domain’s cert isn’t about to expire over cert-manager any day.

IIRC cert-manager has about three layers of custom resources to conv through when figuring out why a cert isn’t renewing.