[marginalia_nu]

My "homelab" is just debian stable, with a moratorium against containerization. Systemd keeps the services going. Updates are manual. It's a bit incredible how clean and easy such a set-up can be. Like there's almost nothing to say. It's extremely reliable and just keeps ticking.

I actually did use to have a bunch of virtualization and kubernetes and crap, but got rid of it because it ate literally half the systems' resources, in a death by a thousand containers-type of way. There was also a lot of just jank, stuff was constantly breaking and I was always trying to piece together why from half a dozen out-of-date tutorials. Felt like herding cats.

[sgarland]

I've transitioned from a single server running services on baremetal Debian, to running services with Docker on Debian, to running services with Docker on Debian VMs under Proxmox (aka Debian with a repo), to running services on a three-node Kubernetes cluster using k3os (I was using Talos, but it doesn't have Longhorn support yet, and Rook/Ceph was a nightmare).

The baremetal server was the easiest to set up, obviously, but the hardest to maintain. Where did I put that random config? Wait, why did this break when I upgraded X? Switching to Docker (and later Compose) wasn't that difficult, and made maintaining services much easier. Going to k8s has been challenging, mostly because of how I'm doing it - a controlplane and worker node on each physical node in Proxmox. Writing the YAML is, well YAML.

I'm mostly looking forward to not having to keep my VMs updated. I automated a lot of that (VMs are Packer templates) with Ansible, but it's still annoying. Upgrading k8s nodes is a lot easier, IMO.

[ryandrake]

Yea I think I missed the whole web-complexity bus and am probably out of touch, but I still don’t get the use case for docker and containers and kubernets and and orchestration and all that stuff, just for a simple home setup. I serve a tiny web site, email, backups, a NAS and a few other internet services for my family, and my “stack” is vanilla Debian Stable.

Maybe I don’t know what I don’t know, but my setup works for me and I don’t really have any problems maintaining it so I figure why add all the complexity?

It always feels weird to see threads and threads of people talking about dozens of software programs I’ve never even heard of, let alone used. Maybe I’m living in the past but to me a “stack” is: OS, server, database, application. Like LAMP. Wonder when this changed!

It makes me curious about what kinds of stuff people do in their home networks that I never even considered doing.

[ClumsyPilot]

"This is just another "how I installed kubernetes" thing only without any real scale behind it."

It is just a learning project, for me K8S didn't 'click' untill i tried to configure it myself on a few machines.

There is something about doing it om physical real things that aids learning, like you could read all chemistry textbooks in the world but untill you actually try it yourself it's not quite the same

I think it's important to keep you 'precious files/services' and you experiments separate.

[naasking]

> but I still don’t get the use case for docker and containers and kubernets and and orchestration

For docker there's a simple motivating case: some services are difficult to configure securely with minimal permissions, and having a standard docker image provided by people who know what they're doing would be a big net win for security on the internet. There are a lot of poorly configured and insecure http server's out there. Think about how many vulnerable http servers are running on routers.

[bingo-bongo]

I would argue that poorly configured docker images as far bigger issue than insecure http servers: https://www.infoq.com/news/2020/12/dockerhub-image-vulnerabi... Security wasn’t the motivation imho, but being able to package your app with all it’s dependencies, including OS libs and deliver it extremely easy, sure was.

[naasking]

> I would argue that poorly configured docker images as far bigger issue than insecure http servers

I think the number of router vulnerabilities alone refutes this argument. Yes, security wasn't the original motivation, but it could be one good motivation.

[sgt]

It doesn't make sense, but often people who set up K8s at home do it for educational reasons or just having a playground so they are more capable at administering systems like that at work.

[iso1210]

Sure, using K8S in your learning envornment because you're managing 800 different services in your dayjob is fine. Personally I have my learning environment as part of my day job though.

At home I just want things to work, hence a nice simple LAMP

[sgt]

Agreed. I don't even want Kubernetes at work.

[iso1210]

I can see benefits of something like ansible if you have a command-and-control environment where you deploy 300 machines from a central database. Still allows you destroy 300 machines in one single misconfiguration or mishandled error though, ouch. If you have a federated environment where you have many actors all managing their own machines though I don't see the point.

The number of times I have to do something to the 160 machines I'm responsible for is about 3 times a year. Clusterssh does the job in 10 minutes, and inevitably shines a light on some unexpected error somewhere. If I was doing it on a weekly basis though, using ansible would be beneficial.

Personally I don't like ansible's way of throwing everything in a file in /tmp, then running it as root. authlog gets sent to my central syslogs servers so I've got a record of what happened. With ansible I'm never quite sure, and I'd have to look in two different places for logs.

Instead we just have a lightweight phone home agent which reports processes/diskspace/mounts/interfaces/hostname/ownership etc so you can identify who owns a device on your network and what it's responsible for without consulting out of date documentation that relies on humans to do things.

Kubernetes I guess is useful if you're building a cluster of 100 machines scaling and doing the same service because you're handling a million users a day. Most companies don't do that, most services don't need that.

Use the right tool for the right job, and don't assume the stuff google use is the right tool (unless you're building a lab because you want experience to get a job working for google. If you tried that shit on my network you'd be starting your job search far earlier)

[Macha]

I want multiple conflicting versions of Elasticsearch on the same host. My OS packages are built to support a single deployment. I could manually setup users, config files, systemd services, etc. for the second deployment, or I could just run them in containers, which is the lower effort option

[marginalia_nu]

> I want multiple conflicting versions of Elasticsearch on the same host

Why do you actually want this? Even having one ES is a bit of a smell that you're probably over-engineering things.

[Macha]

Dependencies of software I didn't write (e.g. Graylog)

[marginalia_nu]

This is what I'm getting at. That is an enormous overkill for a single server, and exactly the problem with kubernetes: It drives so much complexity that you kind of need this type of heavy-duty enterprise grade solutions even in an area where you really shouldn't need it.

If it's just a single server, you're fine with logrotate and grep.

[sgarland]

I like practicing enterprise at home; it makes me feel more comfortable using them at work, where they're needed.

[Macha]

So first of all: I do have to clarify that I just use bare podman containers deployed by ansible and not a while k8s setup, so I'm defending containers and not k8s. But anyway:

Who said graylog is only logging on that single server? That server is by far the biggest and hence why I run multiple services with big dependencies on it, but I have that server, 2 cloud servers (one running my personal web page and other publically accessible services, one running a couple of game servers) + a couple of Raspberry Pis.

Even if I didn't have all that, a web interface to configure it to email on ssh login or send me a mobile notification that my offsite backups are still happening and/or not happening for devices or similar items would still be a nice feature, even if you personally feel it's too complex or you could write a cron job for each of them.