[commanderjroc]

This feels like a post ranting against SystemD written from someone who likes init.

I understand that K8 does many things but its also how you look at the problem. K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail.

Arguably, this is one problem that is made up of smaller problems that are solved by smaller services just like SystemD works.

Sometimes I wonder if the Perlis-Thompson Principle and the Unix Philosophy have become a way to force a legalistic view of software development or are just out-dated.

[dolni]

I don't find the comparison to systemd to be convincing here.

The end-result of systemd for the average administrator is that you no longer need to write finicky, tens or hundreds of line init scripts. They're reduced to unit files which are often just 10-15 lines. systemd is designed to replace old stuff.

The result of Kubernetes for the average administrator is a massively complex system with its own unique concepts. It needs to be well understood if you want to be able to administrate it effectively. Updates come fast and loose, and updates are going to impact an entire cluster. Kubernetes, unlike systemd, is designed to be built _on top of_ existing technologies you'd be using anyway (cloud provider autoscaling, load balancing, storage). So rather than being like systemd, which adds some complexity and also takes some away, Kubernetes only adds.

[throwaway894345]

> So rather than being like systemd, which adds some complexity and also takes some away, Kubernetes only adds.

Here are some bits of complexity that managed Kubernetes takes away:

* SSH configuration

* Key management

* Certificate management (via cert-manager)

* DNS management (via external-dns)

* Auto-scaling

* Process management

* Logging

* Host monitoring

* Infra as code

* Instance profiles

* Reverse proxy

* TLS

* HTTP -> HTTPS redirection

So maybe your point was "the VMs still exist" which is true, but I generally don't care because the work required of me goes away. Alternatively, you have to have most/all of these things anyway, so if you're not using Kubernetes you're cobbling together solutions for these things which has the following implications:

1. You will not be able to find candidates who know your bespoke solution, whereas you can find people who know Kubernetes.

2. Training people on your bespoke solution will be harder. You will have to write a lot more documentation whereas there is an abundance of high quality documentation and training material available for Kubernetes.

3. When something inevitably breaks with your bespoke solution, you're unlikely to get much help Googling around, whereas it's very likely that you'll find what you need to diagnose / fix / work around your Kubernetes problem.

4. Kubernetes improves at a rapid pace, and you can get those improvements for nearly free. To improve your bespoke solution, you have to take the time to do it all yourself.

5. You're probably not going to have the financial backing to build your bespoke solution to the same quality caliber that the Kubernetes folks are able to devote (yes, Kubernetes has its problems, but unless you're at a FAANG then your homegrown solution is almost certainly going to be poorer quality if only because management won't give you the resources you need to build it properly).

[dolni]

Respectfully, I think you have a lot of ignorance about what a typical cloud provider offers. Let's go through these each step-by-step.

> SSH configuration

Do you mean the configuration for sshd? What special requirements would have that Kubernetes would help fulfill?

> Key management

Assuming you mean SSH authorized keys since you left this unspecified. AWS does this with EC2 instance connect.

> Certificate management (via cert-manager)

AWS has ACM.

> DNS management (via external-dns)

This is not even a problem if you use AWS cloud primatives. You point Route 53 at a load balancer, which automatically discovers instances from a target group.

> Auto-scaling

AWS already does this via autoscaling.

> Process management

systemd and/or docker do this for you.

> Logging

AWS can send instance logs to CloudWatch. See https://docs.aws.amazon.com/systems-manager/latest/userguide....

> Host monitoring

In what sense? Amazon target groups can monitor the health of a service and automatically replace instances that report unhealthy, time out, or otherwise.

> Infra as code

I mean, you have to have a description somewhere of your pods. It's still "infra as code", just in the form prescribed by Kubernetes.

> Instance profiles

Instance profiles are replaced by secrets, which I'm not sure is better, just different. In either case, if you're following best practices, you need to configure security policies and apply them appropriately.

> Reverse proxy

AWS load balancers and target groups do this for you.

> HTTPS

AWS load balancers, CloudFront, do this for you. ACM issues the certificates.

I won't address the remainder of your post because it seems contingent on the incorrect assumption that all of these are "bespoke solutions" that just have to be completely reinvented if you choose not to use Kubernetes.

[throwaway894345]

> I won't address the remainder of your post because it seems contingent on the incorrect assumption that all of these are "bespoke solutions" that just have to be completely reinvented if you choose not to use Kubernetes.

You fundamentally misunderstood my post. I wasn't arguing that you had to reinvent these components. The "bespoke solution" is the configuration and assembly of these components ("cloud provider primitives" if you like) into a system that suitably replaces Kubernetes for a given organization. Of course you can build your own bespoke alternative--that was the prior state of the world before Kubernetes debuted.

[dolni]

That's not really any different for Kubernetes.

You still need to figure out where your persistent storage is.

You still have to send logs somewhere for aggregation.

You have the added difficulty of figuring out cost tracking in Kubernetes since there is not a clear delineation between cloud resources.

You have to configure an ingress controller.

You want SSL? Gotta set that up, too.

You have to figure out how pods are assigned to nodes in your cluster, if separation of services is at all a concern (either for security or performance reasons).

Kubernetes is no better with the creation of "bespoke solutions" than using what your cloud provider offers.

Compare this tutorial for configuring SSL for Kubernetes services to an equivalent for configuring SSL on an AWS load balancer. Is Kubernetes really adding value here?

https://blog.karmacomputing.co.uk/kubernetes-cluster-with-ss... https://aws.amazon.com/premiumsupport/knowledge-center/assoc...

[parasubvert]

Kubernetes is far better for each of the above tasks because it is a consistent approach and set of abstractions rather than looking through the arbitrary "everything store" of the cloud providers. I really don't have any interest in relying on 15 different options from cloud providers, I want to get going with a set of extensible, composable abstractions and control logic. Software should not be tied to the hardware I rent or the marketing whims of said entity.

Yes, there is choice and variety among Kubernetes extensions, but they all have fundamental operational assumptions that are aligned because they sit inside the Kubernetes control and API model. It is a golden era to have such a rich set of open and elegant building blocks for modern distributed systems platform design and operations.

[throwaway894345]

Well, first of all, note how much shorter your list is than the original. So vanilla Kubernetes is already taking care of lots of things for us (SSH configuration, process management, log exfiltration, etc). Moreover, we're not talking about vanilla Kubernetes, but managed Kubernetes (I've been very clear and explicit about this) so most of your points are already handled.

> You still need to figure out where your persistent storage is.

Managed Kubernetes comes with persistent storage solutions out of the box. I don't know what you mean by "figure out where it is". On EKS it's EFS, on GKE it's FileStore, and of course you can use other off-the-shelf solutions if you prefer, but there are defaults that you don't have to laboriously set up.

> You still have to send logs somewhere for aggregation.

No, these too are automatically sent to CloudWatch or equivalent (maybe you have to explicitly say "use cloudwatch" in some configuration option when setting up the cluster, but still that's a lot different than writing ansible scripts to install and configure fluentd on each host).

> You have the added difficulty of figuring out cost tracking in Kubernetes since there is not a clear delineation between cloud resources.

This isn't true at all. Your cloud provider still rolls up costs by type of resource, and just like with VMs you still have to tag things in order to roll costs up by business unit.

> You have to configure an ingress controller.

Nope, this also comes out of the box with your cloud provider. It hooks into the cloud provider's layer 7 load balancer offering. It's also trivial to install other load balancer controllers.

> You want SSL? Gotta set that up, too. ... Compare this tutorial for configuring SSL for Kubernetes services to an equivalent for configuring SSL on an AWS load balancer. Is Kubernetes really adding value here?

If you use cert-manager and external-dns, then you'll have DNS and SSL configured for every service you ever create on your cluster. By contrast, on AWS you'll need to manually associate DNS records and certificates with each of your load balancers. Configuring LetsEncrypt for your ACM certs is also quite a lot more complicated than for cert-manager.

> Kubernetes is no better with the creation of "bespoke solutions" than using what your cloud provider offers.

I hope by this point it's pretty clear that you're mistaken. Even if SSL/TLS is no easier with Kubernetes than with VMs/other cloud primitives, we've already addressed a long list of things you don't need to contend with if you use managed Kubernetes versus cobbling together your own system based on lower level cloud primitives. And Kubernetes is also standardized, so you can rely on lots of high quality documentation, training material, industry experience, FAQ resources (e.g., stack overflow), etc which you would have to roll yourself for your bespoke solution.

[mst]

Right, I really dislike systemd in many ways ... but I love what it enables people to do and accept that for all my grumpyness about it, it is overall a net win in many scenarios.

k8s ... I think is often overkill in a way that simply doesn't apply to systemd.

[parasubvert]

If you have to manage a large distributed software code base or set of datacenters, Kubernetes is a win in that it provides a consistent, elegant solution to a nearly universal set of problems.

Systemd comparatively feels like a complete waste of time given the heat it has generated for the benefit.

[thethethethe]

> The end-result of systemd for the average administrator is that you no longer need to write finicky, tens or hundreds of line init scripts.

Wouldn't the hundreds of lines of finicky, bespoke Ansible/Chef/Puppet configs required to manage non-k8s infra be the equivalent to this?

[mathw]

In my work, absolutely yes. Using Kubernetes has saved us sooo much nonsense. Yes we have a mix of Terraform and k8s manifests to deploy to Azure Kubernetes Service, but it works out pretty well in the end.

Honestly most of the annoyance is Azure stuff. Kubernetes stuff is pretty joyful and, unlike Azure, the documentation sometimes even explains how it works.

[dolni]

I can't say I have had the same experience.

Kubernetes cluster changes potentially create issues for all services operating in that cluster.

Provisioning logic that is baked into an image means changes to one service have no chance of affecting other services (app updates that create poor netizen behavior, notwithstanding). Rolling back an AMI is as trivial as setting the AMI back in the launch template and respinning instances.

There is a lot to be said for being able to make changes that you are confident will have a limited scope.

[dolni]

Does Kubernetes infrastructure also not require some form of configuration?

Yes, there is a trade off here. You are trading a staggeringly complex external dependency for a little bit of configuration you write yourself.

The Kubernetes master branch weighs in at ~4.6 million lines of code right now. Ansible sits at ~286k on their devel branch (this includes the core functionality of Ansible but not every single module). You could choose not to even use Ansible and just write a small shell script that builds out an image which does something useful in less than 500 lines of your own code, easily.

Kubernetes does useful stuff and may take some work off your plate. It's also a risk. If it breaks, you get to keep both of the pieces. Kubernetes occupies the highly unenviable space of having to do highly available network clustering. As a piece of software, it is complex because it has to be.

Most people don't need the functionality provided by Kubernetes. There are some niceties. But if I have to choose between "this ~500 line homebrew shell script broke" and "a Kubernetes upgrade went wrong" I know which one I am choosing, and it's not the Kubernetes problem.

Managed Kubernetes, like managed cloud services, mitigate some of those issues. But you can still end up with issues like mismatched node sizes and pod resource requirements, so there is a bunch of unused compute.

TL;DR of course there are trade-offs, no solution is magic.

[thethethethe]

Fair, I was just pointing out that there was more to the analogy. Systemd, like init, also requires configuration, though it is more declarative than imperative, similar to k8s. Some people may prefer this style and consider it easier to manage, however, I my opinions here are not that strong

[0xEFF]

Kubernetes removes the complexity of keeping a process (service) available.

There’s a lot to unpack in that sentence, which is to say there’s a lot of complexity it removes.

Agree it does add as well.

I’m not convinced k8s is a net increase in complexity after everything is accounted for. Authentication, authorization, availability, monitoring, logging, deployment tooling, auto scaling, abstracting the underlying infrastructure, etc…

[dolni]

> Kubernetes removes the complexity of keeping a process (service) available.

Does it really do that if it you just use it to provision an AWS load balancer, which can do health checks and terminate unhealthy instances for you? No.

Sure, you could run some other ingress controller but now you have _yet another_ thing to manage.

[randomswede]

Do AWS load balancers distinguish between "do not send traffic" and "needs termination"?

Kubernetes has readiness checks and health checks for a reason. The readiness check is a gate for "should receive traffic" and the health check is a gate for "should be restarted".

[cjalmeida]

If that’s all you use k8s for, you don’t need it.

Myself I need a to setup a bunch of other cloud services for day 2 operations.

And I need to do it consistently across clouds. The kind of clients I serve won’t use my product as a SaaS due to regulatory/security reasons.

[dolni]

Multi-cloud is one of the few compelling use cases I can think of for Kubernetes.

That said, there are relatively few organizations that actually require it.

[rossmohax]

> K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail.

K8S does very simple stateless case well, but anything more complicated and you are on your own. Statefull services is still a major pain especially thus with leader elections. There is not feedback to K8S about application state of the cluster, so it can't know which instancess are less disruptive to shut down or which shard needs more capacity.

[throwaway894345]

> I understand that K8 does many things but its also how you look at the problem. K8 does one thing well, manage complex distributed systems such as knowing when to scale up and down if you so choose and when to start up new pods when they fail.

Also, in the sense of "many small components that each do one thing well", k8s is even more Unix-like than Unix in that almost everything in k8s is just a controller for a specific resource type.