[nyellin]

To everyone saying that Kubernetes is unnecessary, try implementing autoscaling, service discovery, secrets management, and autohealing in a vendor independent way without it.

Of course none of that is necessary for a self hosted home lab, but neither is gitops.

This is a very nice example of how to set stuff up properly.

OP, I would love to see Robusta (https://robusta.dev) as part of this too. It's definitely in line with your vision of automating everything, as it let's you automate the response to alerts and other events in your cluster. (Disclaimer: I'm one of the maintainers)

[dijit]

Oh. I’ve actually done that. It’s not as hard or complicated as you claim.

This was before kubernetes, arguably learning kubernetes is harder than implementing auto discovery and auto healing because you have to learn a lot about cluster semantics, the various network providers, service accounts, role based access control, integrations with secrets manager or the very disparate implementations of custom operators which may operate in your environment.

Kubernetes is a platform, I don’t doubt it solves a lot of problems. But it’s a complicated wee beastie, and I say that as a person who did it the easy way and used GKE on Google Cloud

[nyellin]

I don't know how you implemented it, but the big innovation of kubernetes was doing this with declarative inputs and using control loops for reconciliation instead of edge triggered changes.

It might not matter on a small scale, but on a large scale that's what makes it robust. It's also not trivial to implement a distributed version of this in house at scale. At the very least you need something like etcd at the core

[dijit]

The way we did it is that we registered to a centralised “name” service which was an in memory database.

Failure to register to the name service leads to registration towards the next name service in the list, we ran three- so there were two spares, the library would then keep trying to register to the first name service, this would be our reconciliation loop.

When you request a service you request “traits” through the name service.

Not “trivial” but even though it was implemented in C++ on windows it was much less complicated than kubernetes discovery systems, which there are many ways to integrate (env-var, dns, api) and some should or must be disabled for security reasons (eg. The environment variables that display all services) or can change depending on RBAC and service account.

We deployed about 300,000 or so services with this infrastructure and it’s been running in production since 2015.

(Original product release was supposed to be 2013 but due to the client changes it was delayed, not due to the backend. This is Video Games unfortunately).

I’m not like, “hard” on these opinions, but people talk about kubernetes as if it’s the only way of solving these problems, when it can be the case that you have to learn a lot about kubernetes before you can actually do what you need to do; if you already understand distributed systems then it’s a bigger effort to understand kubernetes than would be to implement a distributed system that solves your needs from scratch

[nyellin]

Very cool and thanks for sharing.