[zmmmmm]

> all of those choices have been made and agreed upon

Have they really? I have a few apps deployed on k8s and I feel like every time I need something, it turns out it doesn't do that and I'm into some exotic extension or plugin type ecosystem.

Something as simple as service autoscaling (this was a few years ago) was an adventure into DIY. Moving from google cloud to AWS was a complete writeoff almost - just build it again.

I'm sure it captures some layer of abstraction that's useful but my personal experience is it seems very thin and elusive.

[chillfox]

Yep, this is my main problem with k8s, it really feels like none of the choices have been made, it's all chose and configure components.

[hedora]

This, and because of that, claiming your app "runs in kubernetes" is completely meaningless.

Concretely: Take your app. With one button click, or apt-get install ??? on all your machines, configure k8s. Now, run your app.

The idea that this could work has been laughable for any k8s production environment I've seen, which means you can't do things like write automated tests that inject failures into the etcd control plane, etc.

(Yes, I know there are chaos-monkey things, but they can't simulate realistic failures like kernel panics or machine reboots, because that'd impact other tenants of the Kubernetes cluster, which, realistically, is probably single tenant, but I digress..)

If your configuration is megabytes of impossible to understand YAML, and is also not portable to other environments, then what's the point?

(I understand the point for vendors in the ecosystem: People pay them for things like CNI and CSI, which replace Linux's network + storage primitives with slower, more complicated stuff that has worse fault tolerance semantics. Again, I digress...)

[philipallstar]

> If your configuration is megabytes of impossible to understand YAML, and is also not portable to other environments, then what's the point?

If almost all your configuration is about getting Kubernetes set up, and not about your application setup inside Kubernetes, there probably isn't a point. But being able to use roughly the same config inside different Kubernetes is quite good.

[hedora]

But I've never seen portable kubernetes configs (except for vendor software that probably wouldn't be needed outside of kubernetes).

If you just tell kubectl to dump your pod configs, then load them on some other cluster, that definitely won't work.

If you use the management software that generated the pod setup somewhere else, that probably won't work either because the somewhere else is going to be missing the CSI and CNI you targeted. Even if those match, it'll be missing the CRDs. God help you if you want to run two programs on one Kubernetes, and there's a CRD versioning conflict in their two dependency sets.

[esseph]

> Moving from google cloud to AWS was a complete writeoff almost - just build it again.

Yep. Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons. Could someone that is familiar with one pick up on the other? Sure! But there are gotchas. And then kubernetes on prem adds the hardware lifecycle piece, and potential data locality issues, etc.

[physicles]

There are differences across vendors, but there’s a way to build with k8s where the benefit far outweighs the cost.

We run a bunch of services in two very different cloud vendors (one of which used to be DIYed with kubeadm), and also on dev machines with k3s. Takes a while to figure this out and to draw the kustomize boundaries in the right place, but once you do, it’s actually really nice.

Two things work in our favor:

- we’ve been at this for around 8 years, so we didn’t have to deal with all the gotchas at once

- we aggressively avoid tech that isn’t universal (so S3 is OK, but SQS or DynamoDB is not; use haproxy instead of ingress controllers; etc)

[philipallstar]

> Kubernetes is not just kubernetes when moving between clouds, it becomes a very opinionated product (for better or worse) with lots of vendor addons.

I think this is gradually getting better. Networking with Gateways is better than with Ingress in this sense. Things like autoscaling groups need to get better, as they are (or were a couple of years ago) very bespoke.

[chaos_emergent]

I wouldn’t really call it “DIY” per se, k8s has the resource API and you can create whatever scaling policies you want to with it, but I do see how that’s not obvious when it’s advertised as ‘batteries included’