[fen4o]

I understand their rationale. We manage thousand Kubernetes clusters and end-users can find lots and lots of creative way to shoot themselves in the foot:

- I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store.

- I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I never clean those up and my Node simply fills up. Also I add some additional docker networks that break Pod to Pod networks.

- Istio is nice - why we don't add automatic injection for Pods in all namespaces? Including kube-system? And then they brick kube-proxy and the cluster stops working.

- I can use validating webhooks for better security? Lets watch on all resources. To keep it more secure lets set the failure policy of the webhook to Fail, so we never admit any modification without the apiserver to make a call to out webhook. Whats that? My single replica webhook has was evicted from the Pod (we didn't add any resource requests and limits) and now it cannot even be created or scheduled because kube-controller-manager and kube-scheduler cannot update their lease and they lost leadership and now are idling, effectively bricking the entire cluster.

Google would reduce the pain points with this change, however they would still face countless other issues with Kubernetes.

[endymi0n]

I fully agree with your points and would sum them up as "Kubernetes has a steep learning curve, a (quite) large interface and ample opportunities to shoot yourself into the foot with it" (plus, they're very funny).

However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of.

If you actually go all-in and fit your application to the principles of Kubernetes-native applications (instead of the other way around), then it works nothing short to amazing.

We're running 120 microservices in GKE and the difference to our custom-built setup before is night and day. I let my Infra team go surfing together for two weeks because without changes it flies mostly on autopilot.

Let's not kid ourselves, distributed computing is _hard_ and Kubernetes is a testament to that. I'm not saying it can't be made more accessible by further standardization, but there are fundamental limits to how easy it can be made.

Which by the way is leading to my only pet peeve with it: I feel most of the complexity of K8S comes from the fact that it got hyped as an enterprise product and then lots of features were built that support shoving your non cloud-native workload into Kubernetes even if it was never designed for it.

If you don't do or need all of that, the amount of interface, complexity and footguns shrinks significantly. Maybe it's time to better pull them apart in the documentation.

[FlyingSnake]

> If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of.

This argument basically sums up to "Developers just need discipline, and stop blaming the tools". While this is a sound argument on paper, the intrinsic complexity of software systems make it hard to pin the blame on developers. BTW This is the same argument Uncle Bob makes which is not so popular with many mainstream developers.

You're right about feature creep in k8s though.

[endymi0n]

I get what you're saying, but my point is a bit more nuanced:

If your goal is to build highly reliable and available services to end users that are secure and scalable with a team of more than 10 engineers, eventually you will run into more than 50% of the concepts in Kubernetes anyway and end up re-inventing them.

Scaling up and down, node draining, finding out whether services are healthy, RBAC, resource distribution, secrets management, service hardening, introspection capabilities, explicit declaration of dependencies and endpoints and many, many more.

My point is: Sure, if your goal isn't that, it doesn't make sense to start out using Kubernetes.

But if at least eventually that's what you need, imho it's way preferable to just learn and apply well proven abstractions instead of reinventing the wheel along the way and end up with a less maintainable, capable and standardized solution you won't find anyone for maintaining.

If I hear about some of the comments here suggesting to "just spinning up docker-compose with Traefik in front" (disclaimer: I really like Traefik), then that reminds me of how some of the ops mess started that I historically had to care for.

[FlyingSnake]

Agreed, the truth usually lies somewhere in between and my point was we can't absolve the tools/ecosystems and put it on squarely on the devs. That definitely doesn't absolve teams and they need to do their homework before jumping on the bandwagon. K8s is great if you know what you're signing up for.

[pojzon]

We are too proud to sometimes admit we simply „are not good enough at something”.

It’s always easier to shift the blame somewhere else.

Thats why some radical but correct concepts are so hard to push.

[machiaweliczny]

IMO fixing BEAM VM to easily work on cluster could be better for distributed systems than k8s.

[YawningAngel]

That would do nothing to help the 99% of users who don't use BEAM VM languages

[jononomo]

They'd have a reason to start using BEAM VM languages, though.

[yebyen]

To a person who isn't using Erlang or ASP.net, suggesting that we should use either of those language packages and it will solve any of our problems without creating a thousand new ones sounds equally non-starterish to me.

To add a counter-example, I have lots of Ruby experience and I've just joined a Go team. I won't tell them to use Ruby, I will just do it where it makes sense and saves us time. (And then we'll have two problems... enter "limiting blast radius")

Point of my counter-example is, I'm extremely skeptical that all the world's problems can be solved by adopting a new mono-culture, whatever it is. There are 100% always gonna be some problems that are better solved in a different language. PHP is the best way to run Wordpress, for example (ok, so it's the only way to run Wordpress, but you get the idea... "Wordpress is the best way to..."), but I've been in high-functioning IT organizations that won't touch that with a ten foot pole, because "it's another language to support, and PHP is icky."

We also got rid of a perfectly fine Wiki in favor of centralized Knowledge-base software for similar reason. "Better to just have one KB. We don't need to be hosting another thing." So the chances of moving everything over to BEAM VM are next to nil, unless you are a product-focused company with just one product, or happen to have an absolute champion leading the effort to migrate all the things. For all the other things, you need to have a consistent answer too.

No tool is one-size-fits-all. Where Kubernetes shines most is under any environment that isn't running a single monolith or building a software monoculture and/or can't manage that for whatever reason (because those are all basic use cases that are frankly easy enough to manage without adding on top the additional complexity of Kubernetes; don't need it, don't use it!) IMHO, diversity in infrastructure is a plus though, and Kubernetes is a technology that it turns out enables this.

[edude03]

The BEAM VM is fantastic but the scope of Kubernetes + docker is completely different.

For example you still need a way to get BEAM onto hosts, still need to manage the OS on the host, still need to setup networking, RBAC etc.

[secondcoming]

Ok, but we have a similar setup that runs on GCE instances. Deploying involves building an image and pushing a button. We don't really have the need for an Infrastructure team.

[dilyevsky]

I once misconfigured iptables and locked myself out of our buildserver. Had to call lab support in a different country. Is Linux too complicated? Joyent famously took down their whole region by rebooting wrong nodes. It’s almost like running distributed networks of supercomputers at scale is hard or something...

[KronisLV]

> Joyent famously took down their whole region by rebooting wrong nodes.

In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident:

GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind

https://www.youtube.com/watch?v=30jNsCVLpAE

[regularfry]

For two systems of equal functionality, the one that allows or encourages fewer footbullets is the better design. Not all complexity is essential.

[dilyevsky]

What kubernetes complexity is non-essential and can be replicated by simpler solution?

[tormeh]

I once did an `apt autoremove` on a custom install of CentOS handed to my team. Apt uninstalled python (and a lot more), and apt depends on python to run, so that was a bummer. The easiest way out was to reinstall the OS.

[el_seano]

These are some delightfully specific hypotheticals.

[antpls]

There is a benefit : fixing each of these issues fixes them for everyone using K8S.

One of the goal of K8S is normalization/standardization of a complex topic to better share knowledge

[runeks]

> I can store anything in a secret? Let's have thousands of cat images.

Why would someone want to store non-secret information as a secret?

[endymi0n]

"A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools"

— Douglas Adams

[JeremyNT]

Because mutable persistence in Kubernetes can be super annoying to manage and people might grasp for whatever lifeline they can find.

If you have a managed object store or even relational database outside of k8s, the thought of storing arbitrary data in secrets probably doesn't come to mind. But if your enterprise spools up a cluster and tells you to use nfs PVCs with no other storage solution, suddenly you might start getting creative.

[young_unixer]

I think it's just a cute way of saying "data". Like saying you're seeding Linux ISOs when you're actually seeding pirated movies on BitTorrent.

[ForHackernews]

What makes you think Kubernetes "secrets" are appropriate for storing secret information? They're not secure (not without adding a bunch of other nonsense on top of them).

[nanis]

> Why would someone want to store non-secret information as a secret?

Top reason given to me by developers: "I don't want to spend time thinking about the distinction."

[iooi]

> - I can run a root Pod? Lets mount the docker socket and start building images with it.

Just in case you haven't figured out the proper way to do this, you should use docker:dind.

[edude03]

kaniko is an even better way

[sschueller]

If it didn't have wierd issues when you start stacking in docker file.

[edude03]

like what? I'd like to hear your experiences with it

[raverbashing]

Oh yes secret management with kubectl is needlessly complicated

Sure just put your secret data on a file then we'll use your file name as the key of the secret.

Cronjobs sometimes have weird bugs as well.

A lot of its complexity is due to the fact it's an evolving system, that's fine. But I see that some things end up way more complex or unreliable than it needs due to overengineering or use cases no one needs