[makeitdouble]

> things were simple back then

If you were working in the orgs targeted by k8s, I think it was generally more of a mess. Think about managing a park of 100~200 servers with home made bash scripts and crappy monitoring tools and a modicum of dashboards.

Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.

[mrweasel]

You can manage and reason about ~2000+ servers without Kubernetes, even with a relatively small team, say about 100 - 150, depending on what kind of business you're in. I'd recommend either Puppet, Ansible (with AWX) and/or Ubuntu Landscape (assuming that your in the Ubuntu ecosystem).

Kubernetes is for rather special case environments. I am coming around to the idea of using Kubernetes more, but I still think that if you're not provisioning bare-metal worker nodes, then don't bother with Kubernetes.

The problem is that Kubernetes provides orchestration which is missing, or at least limited, in the VM and bare-metal world, so I can understand reaching for Kubernetes, because it is providing a relatively uniform interface for your infrastructure. It just comes at the cost of additional complexity.

Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. Then it's mostly configuration that needs to be pushed and that simplifies things somewhat.

[Hikikomori]

Meanwhile we manage over 1200 instances with multiple kubernetes clusters with a team of 10, including complex mesh networking and everything else the team does. It might be complex but it also gives you so much for free that you don't have to deal with.

[jitl]

imo if you are on a cloud like aws and using a config management system for mutable infra like puppet you are taking unnecessary complexity and living in the dark ages

> Generally speaking I think people need to be more comfortable with build packages for their operating system of choice and install applications that way. The it's mostly configuration that needs

why, it’s 2025, docker / container makes life so easy

[immibis]

because programmers should be able to use computers

[mlrtime]

No, they should be able to take business requirements and create performant reliable applications.

They should understand CS/CE core fundamentals but they don't need to know how to admin.

[immibis]

You might not make it your day job but you should definitely understand the fundamentals of how your whole stack works. Everything from transistors to eyeballs.

[jitl]

Your original suggestion didn’t sound didactic in nature. I did enough deploying Perl apps that way to consider it a huge waste of time. No thanks!

[mlrtime]

>for their operating system of choice...

Have you been in a company with ~2000+ servers where devs install their apps on these OSs and building packages that refuse to upgrade to the latest OS? I mean even with LTS a 20 year old company may still have 3-4 LTS OSs because that last 5% refuse to or cannot upgrade their application to work with the new OS. Sure you could VM the entire thing, but Docker + K8s removes that completely.

[KaiserPro]

> Have you been in a company with ~2000+ servers where devs install their apps on these OSs and building packages that refuse to upgrade to the latest OS

THats what ld_preload is for. But real talk, if you have 2k servers and you can't package your apps to run on your OS, then you need a different platform team.

We managed 36k servers using fucking salt and perl. We were packaging nvidia drivers and all sorts. One system that everyone used still needed the athena widget set.

But the main point is, if you're using old packages, then you're gonna get hacked. You either need to kill that app, fire that developer or virtualise it and fill out the risk register, and do monthly recovery tests.

Docker allows you to pack in CVEs like no tomorrow. so sure k8s can let you do that, and given the hardly anyone properly enforces namespace isolation, so they can have a service mesh, you can still steal loads of data from a compromised container.

[steve1977]

If developers don't maintain their apps, it doesn't really matter that much how and where you deploy them. With Kubernetes, you just end up with unmaintained Docker images that potentially contain a ton of vulnerabilities.

[jitl]

Yeah but at least the fucked-ness is contained in the app layer and the infra layer can live in a happy and optimized modern world.

Also, intricate linkage between an app and the host OS also means there’s more work involved with upgrading.

[conor-]

But with a containerized app image you can reduce the blast radius of the poorly maintained app compared to running it bare metal on a host with other services. Also you can still maintain base images to patch/try to reduce vulnerability surfaces

[Grimburger]

> You can manage and reason about ~2000+ servers without Kubernetes, even with a relatively small team, say about 100 - 150

Oh wow, so uh... I'm managing around 1000 nodes over 6 clusters, alone. There's others able to handle things when I'm not around or on leave and meticulously updated docs for them to do so but in general am the only one touching our infra.

I also do dev work the other half of the week for our company.

Ask your boss if he needs a hand :)

[doubled112]

This sounds super familiar.

At one job I was the only IT person and we had ~250 plain boring VMs on some bare metal Linux/KVM hosts. No config management. No Kubernetes. I fixed that quickly. There was one other guy capable of taking a look at most of it.

I was also doing the software builds and client releases, client support, writing the documentation for the software, and fixing that software.

I suspect we would have had no problem scaling up with some better tooling. Imagine a team of 150? When people tell me things like that, it sounds more like the solution isn't much of a solution at all.

[mlrtime]

Managed k8s (GKE/EKS) or self admin k8s? If the former, no problem. If you're building your own clusters on raw cloud or bare metal compute I'm skeptical if doing it solo. Kudos either way!

[geodel]

> Ask your boss if he needs a hand :)

Hehe, you lack skill in empire building. You know "leading a team of highly motivated team of 50+ devops engineers". The kind of talent that postpones patching until you are back from vacation. Or deploying config change that needs at least two rollbacks before finally going in.

[mrweasel]

That is actually very impressive :-) We have a small team to just handle the databases, but that's ~200 MariaDB and Oracle instances, and another to do networking.

How many different applications/services are you running?

In any case, absolutely amazing what one person can manage with modern infrastructure.

[vidarh]

When you say 100-150, are you talking about the whole organisation? Not just devops?

Because 100-150 for the devops would be crazy for a mid-sized system like that.

Unless you're managing Windows servers or something.

[pbowyer]

> Otherwise, there will be far easier solutions at lower scale.

Which solutions do you have in mind?

- VPS with software installed on the host

- VPS(s) with Docker (or similar) running containers built on-host

- Server(s) with Docker Swarm running containers in a registry

- Something Kubernetes like k3s?

In a way there's two problems to solve for small organisations (often 1 server per app, but up to say 3): the server, monitoring it and keeping it up to date, and the app(s) running on each server and deploying and updating them. The app side has more solutions, so I'd rather focus on the server side here.

Like the sibling commenter I strongly dislike the configuration management landscape (with particular dislike of Ansible and maintaining it - my takeaway is never use 3rd party playbooks, always write your own). As often for me these servers are set up, run for a bit and then a new one is set up and the app redeployed to that (easier than an OS upgrade in production) I've gone back to a bash provisioning script, slightly templated config files and copying them into place. It sucks, but not as much as debugging Ansible has.

[dangus]

Even after the bash script era, I don’t think the configuration management landscape gets enough discredit for how bad it is. I never felt like it stopped feeling hacked together and unreliable.

E.g., Chef Software, especially after its acquisition, is just a dumpster fire of weird anti-patterns and seemingly incomplete, buggy implementations.

Ansible is more of the gold standard but I actually moved to Chef to gain a little more capability. But now I hate both of them.

When I just threw this all in the trash in my HomeLab and went to containerization it was a major breath of fresh air and resulted in getting a lot of time back.

For organizations, of the best parts about Kubernetes is that it’s so agnostic so that you can drop in replacements with a level of ease that is just about unheard of in the Ops world.

If you are a small shop you can just start with something simpler and more manageable like k3s or Talos Linux and basically get all the benefits without the full blown k8s management burden.

Would it be simpler to use plain Docker, Docker Swarm, Portainer, something like that? Yeah, but the amount of effort saved versus your ability to adapt in the future seems to favor just choosing Kubernetes as a default option.

[reissbaker]

Yup. K8s is a bit of a pain to keep up with, but Chef and even Ansible are much more painful for other reasons once you have more than a handful of nodes to manage.

It's also basically a standard API that every cloud provider is forced to implement, meaning it's really easy to onboard new compute from almost anyone. Each K8s cloud provider has its own little quirks, but it's much simpler than the massive sea of difference that each cloud's unique API for VM management was (and the tools to paper over that were generally very leaky abstractions in the pre-K8s world).

[bostik]

To quote an ex coworker: all configuration management systems are broken, in equal measure - just in different fashion. They are all trying to shoehorn fundamentally brittle, complex and often mutually exclusive goals behind a single facade.

If you are in the position to pick a config management system, the best you can do is to chart out your current and known upcoming use cases. Then choose the tool that sucks the least for your particular needs.

And three years down the line, pray that you made the right choice.

Yes, kube is hideously complex. Yes, it comes with enormous selection of footguns. But what it does do well, is to allow decoupling host behaviour from service/container behaviour more than 98% of the time. Combined with immutable infrastructure, it is possible to isolate host configuration management to the image pre-bake stage. Leave just the absolute minimum of post-launch config to the boot/provisioning logic, and you have at least a hope of running something solid.

Distributed systems are inherently complex. And the fundamental truth is that inherent complexity can never be eliminated, only moved around.

[jitl]

with EKS and cloud-init these days i dont find any need to even bake AMIs anymore. scaling / autoscaling so easy now with karpenter to create/destroy nodes to fit current demand. i think if you use kubernetes in a very dumb way to just run X copies of Y container behind an ALB with no funny business it just works.

[jabl]

I have to say I hate ansible too (and puppet and cfengine that I have previously used). But it's unclear to me how containers fix the problems ansible solves.

So instead of an ansible playbook/role that installs, say, nginx from the distro package repository, and then pushes some specific configuration, I have a dockerfile that does the same thing? Woohoo?

[dangus]

I think the major important difference is that a dockerfile can’t really break after you get your deployment artifact, whereas configuration management can fail on your underlying nodes if they aren’t crafted perfectly and cause post-deployment failures.

Other issues like secrets and environment management is something I find way more annoying using a tool like Chef.

Try doing a chef policyfile bootstrap that gets some secrets using its own built in chef vault. You can’t do it without wild workarounds because the node isn’t granted access to secrets until it becomes a registered node, and it doesn’t register until a chef client run completes successfully. It’s a really dumb catch-22 design.

The solution is “just use a big 3 cloud secrets vault or Hashicorp vault” and that’s fine but it’s really strange that the tool can’t handle something so simple on its own.

[fragmede]

You use docker to create a thing on your laptop that you know is good and works, then you send the Dockerfile file in to the system and that thing is a static blob of bits. Ansible/puppet/chef/cfengine modify a live thing from one state to another. Sure, you can use qcow vm disk images and vm snapshots to achieve the same thing, but it’s a lot more cumbersome and feels slow and yuckier, and no one packaged it up into a neat little tool that got popular (which is to say, Vagrant is awesome but slow, so docker won out).

[vidarh]

I managed 1000+ VMs without k8s with an orchestrator that less code than most k8s manifests I've had to work with since.

I fully accept that there are sizes and complexities where k8s is a reasonable choice, and sometimes it's a reasonable choice because it's easier to hire for, but the bar should be a lot higher than what it currently is.

It's a reason why I'm putting together alternatives for those of my clients who wants to avoid the complexity.

[PunchyHamster]

> If you were working in the orgs targeted by k8s, I think it was generally more of a mess. Think about managing a park of 100~200 servers with home made bash scripts and crappy monitoring tools and a modicum of dashboards.

We have Configuration Management systems like Puppet in mature enough state for over a decade now.

I haven't installed server manually or "with handmade scripts" in good 12 years by now.

We have park of around 100-200 servers and actually managing hardware is tiny part of it

> Now, k8s has engulfed a lot more than the primary target, but smaller shops go for it because they'r also hoping to hit it big someday I guess. Otherwise, there will be far easier solutions at lower scale.

K8S is popular because it gives developers a lot of power to deploy stuff, without caring much at underlying systems, without bothering ops people too much. Cloud-wise there is a bunch of native ways to just run a few containers that don't involve it but onprem it is nice way to get a bit faster iteration cycle on infrastructure, even if complexity cost is high.

It is overkill for I'd imagine most stuff deployed in K8S and half of deployments are probably motivated by resume padding rather than actual need.

[Stranger43]

I think you underestimate what can be done with actual code because the devops industry seems entirely code averse and seem to prefer a "infrastructure as data" paradigm instead and not even using good well tested/understood formats like sql databases or even object storage but seems to lean towards more fragile formats like yaml.

yes the possix shell is not a good language which is why thinks like perl, python and even php or C got widely used but there is a intermediate layer with tools like fabric(https://www.fabfile.org/) solving a lot of the problems with the fully homegrown without locking you into the "Infrastructure as(manually edited) Data" paradigm that only really works for problems of big scale and low complexity which is exactly the opposite of what you see in many enterprise environments.

[arkh]

> Think about managing a park of 100~200 servers with home made bash scripts and crappy monitoring tools and a modicum of dashboards.

Not even that. One repository I checked this week had some commits which messages were like "synchronize code with what is on production server". Awesome. And that's not counting the number of hidden adhoc cronjobs on multiple servers.

Also as a dev I like having a pool of "compute" where I can decide to start a new project whenever instead of having to ask some OPS team for servers, routing, DNS config.

[cess11]

I've managed a couple of hundred virtual servers on vCenter with Ansible. It was fine. Syslog is your friend.