Hacker News new | ask | show | jobs
by NickBusey 738 days ago
I am currently streaming my work on migrating HomelabOS from a docker-compose based implementation to a k8s (actually k3s) based implementation. With Longhorn backing it, I'm pretty excited by the possibilities. It will stay as generic as HomelabOS is, so it will be deployable on anything from a pi on up to big cloud machines.

I won't be doing any 3d printing or anything like that though, so what the author here is doing looks fun in it's own right!
1 comments
s3rg4fts 738 days ago
Wasn't aware of HomelabOS, looks pretty interesting!

How is your Longhorn performing? I tried setting it up on my nodes but with Gigabit networking and possibly the Pis pretty average CPUs I would get pretty awful performance both in distributed volumes (with replicas etc) but also on strict-local ones (for reasons I haven't yet figured out).

I am now considering using something like https://github.com/rancher/local-path-provisioner, since I mainly intend to use Longhorn for DBs that handle fault tolerance / backups etc on their own.

link
psini 737 days ago
My experience with longhorn, or rook/ceph, or any other StorageClass solution that enables read write many is that eventually, despite all the "self-healing" and "highly available" claims, it all explodes at some point.

Volumes become unmountable, pools stay yellow forever, desynchronization happens... Perhaps (most probably even) I just don't know how to configure them properly, but maybe that is telling of the difficulty to operate/maintain these solutions.

What I took away from this is to try my most to never deploy anything that requires shared persistent volumes. If I need something stateful, it needs to speak to a database or S3 backend, or something that handles redundancy at some other level than filesystem. If I really really need to have a local volume, I'll use local-path-provisoner like you said, which means pinning the pod to a single node, but really that is a concession I am willing to make to not deal with ceph/rook.

Great writeup, it's like I'm reading about my own journey managing kubernetes clusters!

Best of luck

link
NickBusey 738 days ago
I haven't gotten to performance testing just yet, that should happen pretty soon. Thanks for the local-path-provisioner recommendation, I'll definitely give it a go!
link
s3rg4fts 738 days ago
Good luck! This is pretty useful for benchmarks https://github.com/longhorn/kbench?tab=readme-ov-file
link