[xyst]

I suppose this is fine for a local machine setup. But, I would rather setup it a handful of VMs in a k8s cluster.

Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers). Able to not only containerize my workloads but also evacuate workloads to different workers when I need to take down the server for maintenance (os updates, moving, kubectl upgrades).

I had actually planned to setup another remote cluster in my parents home (800 miles away), but ended up 86’ing that because their residential internet is the absolute worst.

Currently cluster only accessible when behind VPN or on local network. Haven’t setup proper authN/authZ controls yet.

[bkiran]

To each their own. When K8's is managed it's awesome.

I would like to do without the headache of dealing with K8's installation or some orchestrator layer. I can "schedule" my on application instances at the size I'm working with.

Also, I run this setup on cloud vm(multiple actually). So it's not restricted to running on a single machine running in a closet.

[yjftsjthsd-h]

> Currently, run my own k8s cluster with 20 worker nodes (basically just VMs on a few computers).

Multiple worker VMs per physical host? Why?

[xyst]

Mostly for easier segregation of workloads. Some of the IoT shit that runs on this cluster I segment it off through k8s and network policies. Also most workloads wouldn’t need to use all cores or memory on that machine (one machine has 128G)

[yjftsjthsd-h]

> Mostly for easier segregation of workloads. Some of the IoT shit that runs on this cluster I segment it off through k8s and network policies.

Depending on your threat model that could make sense.

> Also most workloads wouldn’t need to use all cores or memory on that machine (one machine has 128G)

What? Limiting CPU or memory use of a pod is one of kubernetes' core competencies; just tell it what each thing needs and it handles all that for you.

[Havoc]

There is probably room for both. Planning to use a nix server as storage & DBs for my k8s cluster

[bkiran]

Nice, would love to know how it goes. What kind of storage and DB's are you planning to work with?

[Havoc]

Mostly trying to get rid of longhorn. I've found it to be a continuous source of troubles w/ etcd sync & IO issues. Current iteration of cluster is all optane so might give longhorn another go, but still want to move storage off cluster.

Especially source repo needs to live on some sort of striped zfs array. Ordered one of these quad nvme NAS things [0] so that's probably going to be storage. Either nix or proxmox...not decided.

Also still a bit fuzzy on what best game plan on PVs is. Minio/s3 or nfs appear to be options.

DB...just the usual suspects...mainly postgres for gitea I think. Mongo for dev stuff. Used to vanilla proxmox/docker/lxc so this is all unchartered territory for me.

[0] https://vi.aliexpress.com/item/1005007188570776.html

[bkiran]

I could never get around on longhorn or any of the other "lightweight" K8 distros. I guess that's why I spent time setting up NixOS and writing this post.

Okay nice, seems like various storage services for a home lab setup. Seems like a cool project. Especially if you can distribute it across all those NAS's