[otter-in-a-suit]

> none single-thread performance (relatively)

I find horizontal scaling with many smaller cores and lots of memory more impactful for virtualization workloads than heavy single core performance (which, fwiw, is pretty decent on these Xeon Golds).

The biggest bottleneck is I/O performance, since I rely on SAS drives (since running full VMs has a lot of disk overhead), rather than SSDs, but I cannot justify the expense to upgrade to SSDs, not to mention NVME.

> Those setups always pure "home-lab" because it's too small or macgyvered together for anything, but the smallest businesses...where it will be an overkill.

That is a core part of the hobby. You do some things very enterprise-y and over-engineered (such as redundant PSUs and UPSs), while simultaneously using old hard drives that rely on your SMART monitor and pure chance to work (to pick 2 random examples).

I also re-use old hardware that piles up around the house constantly, such as the Pi. I commented elsewhere that I just slapped an old gaming PC into a 4U case since I want to play/tinker with/learn from GPU passthrough. I would not do this for a business, but I'm happy to spend $200 for a case and rails and stomach an additional ~60W idle power draw to do such. I don't even know what exactly I'll be running on it yet. But I _do_ know that I know embarrassingly little about GPUs, X11, VNC, ... actually work and that I have an unused GTX 1080.

Some of this is simply a build-vs-buy thing (where I get actual usage out of it and have something subjectively better than an off the shelf product), others is pure tinkering. Hacking, if you will. I know a website that usually likes stuff like that.

> You're not going to learn much about k8s from that

It's possible you and I learn things very differently then (and I mean this a lot less snarky than it sounds). I built Raft from scratch in Scala 3 and that told me a lot about Raft and Scala 3, despite being utterly pointless as a product (it's on my website if you care to read it). I have the same experience with everything home lab / Linux / networking - I always learn something new. And I work for a networking company...

[0x457]

> It's possible you and I learn things very differently then (and I mean this a lot less snarky than it sounds). I built Raft from scratch in Scala 3 and that told me a lot about Raft and Scala 3, despite being utterly pointless as a product (it's on my website if you care to read it). I have the same experience with everything home lab / Linux / networking - I always learn something new. And I work for a networking company...

Building k8s from scratch, you're going to learn how to build k8s from scratch. Not how to operate and/or use k8s. Maybe you will learn some configuration management tool along the way unless your plan is to just copy-paste commands from some website into terminal.

> find horizontal scaling with many smaller cores and lots of memory more impactful for virtualization workloads than heavy single core performance (which, fwiw, is pretty decent on these Xeon Golds).

Yeah, if you run a VM for every thing that should be a systemd service, it scales well that way.

[christphrdunder]

> should be a systemd service

"should be" according to your goals. "should not be" according to mine:

1. run untrusted code in a reasonably secure way. i don't care how many github stars it has, i'm not rawdogging it. nor is my threat model mossad, so it doesn't have to be perfect. but systemd's default security posture is weak, hardening it is highly manual (vs. "run a VM"), and must be done per service to do properly (allowlist syscalls, caps, directory accesses, etc.).

2. minimize cost. it's orders of magnitude less costly for most people with the skills to run a homelab to:

   a) spend 50% more for compute & storage than a single hour of handwriting & tuning reasonably secure systemd services, wiring up dependencies, etc.

   b) build a backup and migration strategy once and reuse it for everything. this is technically possible with systemd, too, of course, but way more costly to setup.

   c) one single universal solution. practically everything will run in a VM. this is not true for systemd, esp. isolated systemd services.

if you want to optimize for "learn how to configure systemd", "learn how to hyperoptimize cpu usage", or whatever it may be then great. if other people aren't, they're not necessarily wrong, they may be choosing different tradeoffs. understanding this is an essential step in maturing as an engineer and human being. i truly mean this as encouragement - not rebuke. Otherwise i wouldn't have paid the relatively high cost in time to write it afterall :)

[udev4096]

OP is right (not surprising, given he's a platform engineer with decades of experience unlike you, a nobody). You are projecting your extremely flawed and naive approach of learning k8s. If all you could think of is "copy-paste configs" then sure, you are definitely NOT gonna learn anything. You're just trying to act smart with absolute hatred towards people running k8s outside of businesses

[0x457]

> (not surprising, given he's a platform engineer with decades of experience unlike you, a nobody)

Also, an engineer with experience? I'm calling out over-engineering for the sake of over-engineering.