[itsnowandnever]

etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd. much of the "secret sauce" of kubernetes is the "watch etcd" logic that "watches" desired state and does the cybernetic loop to bring the observed state adhere to the desired state.

[trenchpilgrim]

The API and controller loops are the point of k8s. etcd is an implementation detail and lots of clusters swap it out for something else like sqlite. I'm pretty sure that GCP and Azure are using Spanner or Cosmos instead of etcd for their managed offerings.

[alphabettsy]

Yep. K3s can use SQLite or Postgres.

[itsnowandnever]

not exactly a fair assessment since neither of those were out and/or available to the kubernetes team at the time. sure, some things at many times from now into eternity may be or become better suited for the kubernetes data plane but at the time if etcd wasn't used there would be no kubernetes today

[trenchpilgrim]

The Kubernetes team chose etcd specifically because they were trying to replace Borg's master/slave database at Google. Nothing about Kubernetes requires etcd; the team was trying to solve a Google-internal problem with it (and in the end, didn't gain traction within Google.) k3s uses sqlite by default which was an option at the time, other clusters today use PostgreSQL.

Have you looked at the etcd keys and values in a Kubernetes cluster? It's a _remarkably_ simple schema you could do in pretty much any database with fast prefix or path scans.

[moondev]

Upstream kubernetes literally requires etcd. Anything that changes that is a fork

[trenchpilgrim]

There are many forms of Kubernetes that are not using etcd that are certified by the CNCF as conformant (https://www.cncf.io/training/certification/software-conforma...). The version in the kubernetes/kubernetes repository is a reference implementation, intended to be customized by the community.

[jauntywundrkind]

The API server is the thing. It so happens that the API server can mostly be a thin shell over etcd. But etcd itself while so common is not sacrosanct.

https://github.com/k3s-io/kine is a reasonably adequate substitute for etcd. sqlite, MySQL, PostgreSQL can also be substituted in. Etcd is from the ground up built to be more scale-out reliable, and that rocks to have baked in. But given how easy it is to substitute etcd out, I feel like we are at least a little off if we're trying to say "etcd is also the entire point of k8s" (the APIserver is)

[dmlittle]

It's been a while since I've checked this but a few years ago we tried to limit test kine on a large-ish cluster and it performed pretty poorly. It's fine for small clusters but the way they have to implement the watch semantics makes it perform poorly (at least this was the case a few years ago).

[jauntywundrkind]

Agreed. The subscriptions really is a huge huge part of the magic, and it's a weakpoint of Kine. Thanks for chiming in.

Ideally, i'd love to see a database specific offering. Use postgres async replication (ideally somehow sharded so there's not a single consumer node) to some fan out system that's doing all the watching.

But etcd mostly does the job, seems unlikely to be going anywhere. It's be cool though.

[itsnowandnever]

that's fair but that 99% of all apiserver deployments in the world have the same standard boilerplate footprint is a large part of why it became so ubiquitous. that people running it locally don't have to make any decisions about how to deploy which database or why to use this one over that one... and that's also the same situation in production so people doing stuff in dev aren't punched in the face by an exponentially more complex system in production is huge.

[geoctl]

Is it? I honestly kinda believe that etcd is probably the weakest point in vanilla k8s. It is simply unsuitable for heavy write environments and causes lots of consistency problems under heavy write loads, it's generally slow, it has value size constraints, it offers very primitive querying, etc... Why not replace etcd altogether with something like Postgres + Redis/NATS?

[itsnowandnever]

that touches on what I consider the dichotomy of k8s: it's a really scalable system that makes it easy to spin up a cluster locally on your laptop and interact with the full API locally just like in prod. so it's a super scalable system with a dense array of features. but paradoxically most shops won't need the vast majority of k8s features ever and by the time they scale to where they do need a ton of distributed init features they're extremely close to the point where they'd be better served by a bespoke system conceived from scratch in house that solves problems very specific to the business in question. if you have many thousands of k8s nodes, you're probably in the gray area of if using k8s is worth it because the loop of k8s will never be as fast as a centralized push control plane vs the k8s pull/watch control plane. and naturally at scale that problem will only compound

[pas]

but it's also standard, you can hire for it, outsource it, etc.

and it's pretty modular too, so it can even serve as the host for the bespoke whatever that's needed

though I remember reading the fly.io blog post about their custom scheduler/allocator which illustrates nicely how much of a difference a custom in-house solution makes if works well

[trenchpilgrim]

The other draw: Because k8s is open, you can easily hire employees, contractors, consultants and vendors and have them immediately solve problems within the k8s ecosystem. If you run a bespoke system, you have to train engineers on the system before they can make large contributions.

[varispeed]

> Why not replace etcd altogether with something like Postgres + Redis/NATS?

Holy Raft protocol is the blockchain of cloud.

[trenchpilgrim]

You can do leader election without etcd. The thing etcd buys you is you can have clusters of 3, 5, 7 or 9 DB nodes and lose up to 1, 2, 3, or 4 nodes respectively. But honestly, the vast majority of k8s users would be fine with a single SQL instance backing each k8s cluster and just running two or more k8s clusters for HA.

k3s doesn't require etcd, I'm pretty sure GKE uses Spanner and Azure uses Cosmos under the hood.

[cyberax]

> etcd is also the entire point of k8s. that it's a single self-contained framework and doesn't require an external backer service. there is no kubernetes without etcd.

Sorry, this is just BS. etcd is a fifth wheel in most k8s installations. Even the largest clusters are better off with something like a large-ish instance running a regular DB for the control plane state storage.

Yes, etcd theoretically protects against any kind of node failures and network partitions. But in practice, well, nobody really cares about the control plane being resilient against meteorite strikes and Cthulhu rising from the deeps.

[itsnowandnever]

that's not my point - my point is it would not have gotten the adoption it has without etcd and the fact that it was resilient and scalable out of the box

[cyberax]

It probably would have gotten even faster adoption if it used a saner embedded DB.