Hacker News new | ask | show | jobs
by mountainriver 563 days ago
Manually updating k8s clusters is a huge tradeoff. I can’t imagine doing that to save a couple bucks unless I was desperate
4 comments

I dunno, I've had to spend like two or three hours each month on updating mine for its entire lifetime (of over 5 years now), and that includes losing entire nodes to hardware failure and spinning up new ones.

Originally it was ansible, and so spinning up a new node or updating all nodes was editing one file (k8s version and ssh node list), and then running one ansible command.

Now I'm using nixos, so updating is just bumping the version number, a hash, and typing "colmena apply".

Even migrating the k8s cluster from ansible to nixos was quite easy, I just swapped one node at a time and it all worked.

People are so afraid of just like learning basic linux sysadmin operations, and yet it also makes it way easier to understand and debug the system too, so it pays off.

I had to help someone else with their EKS cluster, and in the end debugging the weird EKS AMI was a nightmare and required spending more time than all the time I've had to spend on my own cluster over the last year combined.

From my perspective, using EKS both costs more money, gives you a worse K8s (you can't use beta features, their ami sucks), and also pushes you to have a worse understanding of the system so that you can't understand bugs as easily and when it breaks it's worse.

if the "couple of bucks" ends up being the cost of an entire team, then hire a small team to do it.

Then get mad at them because they don't "produce value", and fold it into a developers job with an even higher level of abstraction again. This is what we always do.

We at https://syself.com have made a platform with "one-click updates". 100% vanilla Kubernetes on Hetzner.
The "couple bucks" in my experience were difference between viable business and bankrupt one - including time spent on maintaining k8s!