[mrweasel]

You can manage and reason about ~2000+ servers without Kubernetes, even with a relatively small team, say about 100 - 150, depending on what kind of business you're in. I'd recommend either Puppet, Ansible (with AWX) and/or Ubuntu Landscape (assuming that your in the Ubuntu ecosystem).

Kubernetes is for rather special case environments. I am coming around to the idea of using Kubernetes more, but I still think that if you're not provisioning bare-metal worker nodes, then don't bother with Kubernetes.

The problem is that Kubernetes provides orchestration which is missing, or at least limited, in the VM and bare-metal world, so I can understand reaching for Kubernetes, because it is providing a relatively uniform interface for your infrastructure. It just comes at the cost of additional complexity.

Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. Then it's mostly configuration that needs to be pushed and that simplifies things somewhat.

[Hikikomori]

Meanwhile we manage over 1200 instances with multiple kubernetes clusters with a team of 10, including complex mesh networking and everything else the team does. It might be complex but it also gives you so much for free that you don't have to deal with.

[jitl]

imo if you are on a cloud like aws and using a config management system for mutable infra like puppet you are taking unnecessary complexity and living in the dark ages

> Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. The it's mostly configuration that needs

why, it’s 2025, docker / container makes life so easy

[immibis]

because programmers should be able to use computers

[mlrtime]

No, they should be able to take business requirements and create performant reliable applications.

They should understand CS/CE core fundamentals but they don't need to know how to admin.

[immibis]

You might not make it your day job but you should definitely understand the fundamentals of how your whole stack works. Everything from transistors to eyeballs.

[jitl]

Your original suggestion didn’t sound didactic in nature. I did enough deploying Perl apps that way to consider it a huge waste of time. No thanks!

[immibis]

Automate it with a shell script

[mlrtime]

>for their operating system of choice...

Have you been in a company with ~2000+ servers where devs install their apps on these OSs and building packages that refuse to upgrade to the latest OS? I mean even with LTS a 20 year old company may still have 3-4 LTS OSs because that last 5% refuse to or cannot upgrade their application to work with the new OS. Sure you could VM the entire thing, but Docker + K8s removes that completely.

[KaiserPro]

> Have you been in a company with ~2000+ servers where devs install their apps on these OSs and building packages that refuse to upgrade to the latest OS

THats what ld_preload is for. But real talk, if you have 2k servers and you can't package your apps to run on your OS, then you need a different platform team.

We managed 36k servers using fucking salt and perl. We were packaging nvidia drivers and all sorts. One system that everyone used still needed the athena widget set.

But the main point is, if you're using old packages, then you're gonna get hacked. You either need to kill that app, fire that developer or virtualise it and fill out the risk register, and do monthly recovery tests.

Docker allows you to pack in CVEs like no tomorrow. so sure k8s can let you do that, and given the hardly anyone properly enforces namespace isolation, so they can have a service mesh, you can still steal loads of data from a compromised container.

[steve1977]

If developers don't maintain their apps, it doesn't really matter that much how and where you deploy them. With Kubernetes, you just end up with unmaintained Docker images that potentially contain a ton of vulnerabilities.

[jitl]

Yeah but at least the fucked-ness is contained in the app layer and the infra layer can live in a happy and optimized modern world.

Also, intricate linkage between an app and the host OS also means there’s more work involved with upgrading.

[conor-]

But with a containerized app image you can reduce the blast radius of the poorly maintained app compared to running it bare metal on a host with other services. Also you can still maintain base images to patch/try to reduce vulnerability surfaces

[Grimburger]

> You can manage and reason about ~2000+ servers without Kubernetes, even with a relatively small team, say about 100 - 150

Oh wow, so uh... I'm managing around 1000 nodes over 6 clusters, alone. There's others able to handle things when I'm not around or on leave and meticulously updated docs for them to do so but in general am the only one touching our infra.

I also do dev work the other half of the week for our company.

Ask your boss if he needs a hand :)

[doubled112]

This sounds super familiar.

At one job I was the only IT person and we had ~250 plain boring VMs on some bare metal Linux/KVM hosts. No config management. No Kubernetes. I fixed that quickly. There was one other guy capable of taking a look at most of it.

I was also doing the software builds and client releases, client support, writing the documentation for the software, and fixing that software.

I suspect we would have had no problem scaling up with some better tooling. Imagine a team of 150? When people tell me things like that, it sounds more like the solution isn't much of a solution at all.

[mlrtime]

Managed k8s (GKE/EKS) or self admin k8s? If the former, no problem. If you're building your own clusters on raw cloud or bare metal compute I'm skeptical if doing it solo. Kudos either way!

[geodel]

> Ask your boss if he needs a hand :)

Hehe, you lack skill in empire building. You know "leading a team of highly motivated team of 50+ devops engineers". The kind of talent that postpones patching until you are back from vacation. Or deploying config change that needs at least two rollbacks before finally going in.

[mrweasel]

That is actually very impressive :-) We have a small team to just handle the databases, but that's ~200 MariaDB and Oracle instances, and another to do networking.

How many different applications/services are you running?

In any case, absolutely amazing what one person can manage with modern infrastructure.

[vidarh]

When you say 100-150, are you talking about the whole organisation? Not just devops?

Because 100-150 for the devops would be crazy for a mid-sized system like that.

Unless you're managing Windows servers or something.