[kev009]

I've helped build two top 10 service provider networks (10s of Tbps). One on FreeBSD, and one on Linux with Kubernetes.

I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work.

But it also seems to come with a magnitude of accidental and ornamental complexity. I would imply the same about microservices versus, say, figuring out your repository, language, and deployment pipelines to provide a smooth developer and operator experience. Too much of this industry is fashion and navel gazing instead of thinking about the core problems and standing behind a methodology that works for the business. Unless google moves its own infrastructure to Kubernetes, then maybe there's something to be had that couldn't reasonably be done otherwise :)

[hggigg]

Agree with this on all points.

We went from a virtualized server model to managed Kubernetes and costs have escalated considerably. The additional complexity and maintenance overheads of Kubernetes are not trivial and required additional staff hires just to keep things ticking. I think the cost so far from moving from two cages in separate datacentres running blades to AWS is approximately a 6x multiplier including staff. This was all driven on the back of "we must have microservices to scale", something we have failed entirely to do. It's a complete own goal.

[wkat4242]

I think it mainly works for startups which can build it as a greenfield and are extremely sensitive to scaling (either they are exploding within 2 years or they're dead within 2 years). So they plan to scale from 0 to millions quickly. For that kinda business it makes sense because once the exploding growth comes it's super easy to scale up and even do that automatically. That's the power of Kubernetes.

For an existing business not in a hyper-growth phase with lots of restrictions and policies and processes and legacy stuff it's just extra work with no gain. It's just a round peg in a square hole. Wrong tool for the job.

[threeseed]

That's because you've moved to AWS which is an expensive cloud.

You could have easily run Kubernetes on your virtualised servers.

[baq]

IME nothing easy about maintaining your own storage cluster!

[threeseed]

Nothing easy in general about running a distributed platform for highly-available applications.

[baq]

True but compute nodes are basically stateless in comparison to anything storage.

[hggigg]

Define easily?

[stackskipton]

Still requires SRE but alot easier than it used to be: https://www.talos.dev/v1.8/

Also, k3s. https://k3s.io/

[threeseed]

Kubernetes has unquestionably been a game changer.

If you look at most enterprises today you will see it deployed everywhere.

And most of the complexity has been abstracted away by the cloud providers so all you're left with is a system that can handle all manner of different applications and deployment scenarios in a consistent way.

[krageon]

> Kubernetes has unquestionably been a game changer.

> If you look at most enterprises today you will see it deployed everywhere.

This doesn't mean it's a game changer. It just means it has a big cargo-cult and people keep using it regardless of whether they need it.

> most of the complexity has been abstracted away by the cloud providers

The abstractions that people add make things more complex, not less. Unless of course you don't care to understand what you're running, which is precisely the problem with the culture around most folks that use kubernetes (and more broadly the US cloud providers as an entity).

[lotharcable]

Kubernetes is one of those things that are only as complicated as you want it to be.

The problem with Kubernetes and complexity is that because it simplifies a lot of things that are a huge PITA to accomplish on a "homemade" server/container setup is that there are a huge number of products and things you can run on kubernetes to "do stuff".

And it is hard for a lot of people and organizations to resist the "oh shiny" aspect. Stuff like "Oh, look I can do network policies and service meshes!" or "Lets create this really complicated and big thing so we can configure all our AWS infrastructure with kubernetes objects! Who cares that a bad commit can destroy the infrastructure the cluster depends on along with our ability to manage any of it!" or "Look we can have lots of namespaces for all our internal orgs and departments, lets make a gigantic centrally managed Kubernetes cluster that will be managed by IT and that everybody will be forced to use at the same time! Putting all our eggs in one basket is a awesome idea!".

K8s sorta removes the barrier of entry that world normally stop people from implementing those sorts of bad ideas.

Otherwise the core vanilla Kubernetes isn't really that complicated compared to most DIY solutions that try to manage large numbers of apps on clusters of systems. Most of the time it ends up a lot more robust and simpler in the right hands.

[krageon]

You're not wrong about the ideal case, but I've never seen the ideal case in any real (read: paid) context. I've seen hobbyists with very solid setups that were well managed, I've made the thing that kubernetes wants to solve by hand and I understand it's not a trivial (that is to say, not a single afternoon of design and implementation) to do.

I guess I'm just really frustrated with seeing the people that get paid to use and promote it be so bad at what they do and I don't like cleaning up after them :)

[threeseed]

> The abstractions that people add make things more complex, not less

Everything in our industry is built on abstractions.

By your logic we shouldn't be using operating systems, compilers, libraries etc and writing our own custom software to manually move applications between different hosts if they go down.

[krageon]

I didn't say this at all. I said you should understand what you run, and if you understand it you realise it adds complexity.

I don't really see how this is a particularly controversial statement and I will say I don't appreciate you trying to reduce it to some ridiculous extreme. You didn't add anything to the discussion and it's tragic, because you could have made a nice point about system boundaries, levels of abstraction and perhaps information overload.

That would have made a great point about why abstraction is sometimes necessary. Then we could have spoken about what makes good and bad abstractions. Perhaps someone who came upon this thread later could have learned something and we'd have made the site a better place together.

[martinbaun]

You're probably going to be downvoted to oblivion but I 100% agree with you. It seems like they tried to remove all the complexity, and just made new complexity.

[stackskipton]

Modern systems are complex generally. If it's not Kubernetes with YAML files, it's a bunch of servers in VMware which is own ball of wax followed by extensive Ansible/Puppet/Chef setup and SREs who keeping entire architecture in their head since they don't have time to write it down.

Obviously, there is exact opposite with stuff like fly.io but they can be extremely constraining.

[martinbaun]

Yeah, I really liked Heroku and I used it for so many years, but it also killed itself kinda. It could never reduce prices as all machines "was the same" and thus a reduction in price meant lose of revenue and now they're just super expensive comparably.

Now I just dump Caddyserver and install things bare metals servers. I avoid all dependencies like the plague

[krageon]

You will be happy to learn that it was not in fact downvoted at all. I think this may be the first time I've expressed this sentiment to (mostly) positive feedback and I'm pretty sure it's because it was explained properly this time. That is I think a good thing - the downvotes that came before gave me the impulse to improve, and I was rewarded when I did.

[vilunov]

Kubernetes is absolutely not an accidental complexity. Microservices are not an accidental complexity, and they are not a replacement for a proper repository.

Kubernetes solves administration of a cluster of Linux machines, as opposed to administering a single Linux machine. It abstracts away the concept of a machine, because it automates application scheduling, scaling across different machines, rolling updates of applications, adding/removing machines to the cluster all at the same time. There are no instruments like that for applications, the closest to them are something like Spark and Hadoop for data engineering tasks (not general applications).

Microservices are also used to solve a very specific problem – independent deployments of parts of the system. You can dance with your repository and your code directories as much as you want, if you're not in a very specific runtime (e.g. BEAM VM), you will not achieve independent deployments of parts of your service. The ability to "scale independently" (which tbh is mostly bullshit) is an accidental consequence of using HTTP RPC for microservice communication, which is also not the only way, but it allows reuse of the HTTP ecosystem.

[kunley]

Apart from a sober and common-sense thinking in your comment, I like the term "ornamental complexity" :)