[from-nibly]

K8s is painful to get started, and painful to learn. But once you have it up you can just keep adding stuff to it.

I run a k8s cluster at home. Part of it yes, is to apply my existing skills and keep them fresh. But part of it is that kubernetes can be easier long term.

Ive got magical hard drive storage with rook ceph. I can yoink a hard drive out of my servers and nothing happens to my workloads.

I can do maintenance on one of the servers with 0 down time.

All of my config for what I have deployed is in git.

I manage VMS and kubernetes at work, and im not going to pretend that kubernetes isnt complex, but it's complex up front instead of down the road. VMs run into complexity when things change. I'm sure you can make VMS good but then why not use something like kubernetes, you will have to reinvent a lot of the stuff that's already in kubernetes.

It's a hammer for sure and not everything is a nail, but it can be really powerful and useful even for home labs.

[hiAndrewQuinn]

I don't run k8s at home, but I have worked in k8s-heavy environments and studied it deeply. This is the accurate, nuanced take.

Few but not no people will ever run into problems at the kind of scale k8s operates at. Plus, learning how it "expects" the programs running inside its Pods to behave is kind of like learning how Django or Rails "expect" a web app to work - it's a more complicated style than just writing your own totally custom, hermetically-sealed Python apps for your personal use, sure, but it also comes with a slew of benefits in case you ever do hit that level of scale and want to move over.

Or, maybe you look over the app you're writing and say "Fat chance." In which case you can justify e.g. not making everything an API endpoint, keeping a ton of state mucking about, etc. But I still feel that's an improvement over not even realizing the questions are being asked.

[catdog]

What you also can do is starting with just a single node, incredibly easy to install with e.g. https://k3s.io/. You still have to invest the upfront effort to understand how it works but you can already reap a lot of benefits with a lot less complexity.

Kubernetes does not force you into the distributed systems hell, you can go that route later, or never.

[rendaw]

Kubernetes/k3s on a single node turns what could have been immutable 1-step upgrades into multi-step mutable upgrades, since kubernetes's software itself and all the management components you need are a mutable layer on top of the operating system.

[threeseed]

a) It doesn't have to be mutable. You can easily setup k3s on a single node, install the apps and bake an AMI or equivalent. And using something like ArgoCD or GitOps will ensure that your k8s stack is in sync with a tracked and managed Git repository.

b) In what world is upgrading your entire platform ever a single step. Even for a basic Python app you still have Python itself plus dependencies. And then of course whatever front end web server you're using.

[angio]

You can use Talos linux for an immutable (and tiny) OS.

[cyberpunk]

> K8s is painful to get started

Is that really true anymore? Even self hosting k8s these days (e.g with rke/rke2) is a single yaml file and one command to deploy an entire cluster.. Maybe back when we all used kubespray and networking was more complicated (to the user at least) etc.. But today? I don't think so.

Using a hosted offering is even easier, literally a couple of clicks, a ./gcloud-cli or terraform apply -- again not very hard and all the cloud providers provide you with example code you just need to plug some machine sizes etc into..

Dev setup? Install orbstack and click 'kubernetes' and you're done, your IDE (likely) will automagically pick up your kubeconfig and you can go right ahead creating services, deployments, jobs, whatevers...

[from-nibly]

I'm not talking about setting up a cluster. I'm talking about all the learning you have to do.

[dambi0]

I’m sure there are countless other benefits. But how many layers of abstraction, services and things that need configuring are their compared to basic RAID to get support for magical hard disks that can be yoinked without affecting workloads?

[k8sagainand]

You get an aligned infra layer. You get a great opensource ecosystem (k8s, argocd, git / gitops, helm, helm charts, grafana, prometheus etc.)

You get basic loadbalancing, health checks, centralized and nearly out of the box logging and monitoring and tracing.

You get a streamlined build process (create a container image, have an image build, create your helm chart, done)

Your RAID commment is quite far away of what k8s makes k8s

[grep_name]

> Compared to basic RAID to get support for magical hard disks that can be yoinked without affecting workloads?

These things aren't mutually exclusive though. I've spent the last few years working with kubernetes at work and running a 'simple'(but with tons of containers and weird edge cases / uses) unraid server at home for all of my needs. At some point I flipped over from 'jeez kubernetes is just too much, almost nobody should ever use this' to 'wow I have to migrate 99% of my home services to a cluster, this is driving me nuts.' I haven't quite gotten around to that migration, but I do think that k8s cluster for services / temporary storage / parallel jobs and separate unraid box that runs NFS (and doesn't do much else) is going to be a great setup for a home lab.

[ElectricalUnion]

Aren't disks so large those days that losing a disk almost means you will lose a second disk during resilvering unless that by "basic raid" you're doing not-basic-raid things such as btrfs raid1c3?

[marcosdumay]

> But once you have it up you can just keep adding stuff to it.

I dunno why, but the k8s in my workplace keeps breaking in painful ways. It also has an endless supply of breaking points that makes life painful for anybody that depends on what runs in it, but aren't detected by the people that manage it.

Honestly, that second part is an exclusivity there, but I have never seen people "just keeping adding things to it" on practice.

[from-nibly]

It depends on how well you know k8s and what your stack is. Rancher is an extra complex version of k8s. Longhorn is pretty fragile in my experience, so is canal. But chillum and eks don't really have the same reliability issues in my experience.