[mac-chaffee]

I'm currently trying to convince people that a managed k8s service is not that "simple", and that we can't "just spin up another cluster" without a great deal of operational overhead.

Some of the things that might still be needed in managed k8s instances: better ingress with ingress-nginx, cert-manager, monitoring/logging/alerting, tuning the alerts, integration with company SSO, security hardening.

If it's a multi-tenant cluster: LimitRanges/ResourceQuotas, NetworkPolicies, scripts to manage namespaces and roles, PodSecurityPolicies (or equivalent), onboarding offboarding procedures.

I'm sure you'd need similar things to have a proper production Nomad cluster too, so your point still stands. But at least for EKS/GKE clusters, they're pretty bare-bones.

[busterarm]

As someone with 7-digit spend in GKE/EKS, I will agree with you that it is _anything but simple_.

Your developers aren't going to say that it's simple when Google force upgrades their cluster to a version that deprecates APIs in their yamls for a job they worked on 2 years ago and swiftly forgot about.

Then when you explain to them that Google insists on putting everyone on a force-upgrade treadmill, you can literally watch as the panic sets in on the faces of your engineering team managers/leads.

Nomad is a breeze in comparison to managed K8s.

Everyone that I've talked to that thinks Kubernetes is simple is barely using the thing and could likely save a lot of money and development effort using something like Nomad instead.

[jpgvm]

So instead of using higher level primitives that are widely used and tested and have many simple high quality integrations (external-dns, cert-manager, et al) you would recommend reinventing all of that on Nomad and then calling that "saving a lot of money and development effort"?

Yeah no thanks.

At this point k8s has "won" for all intents and purposes. It has gained critical mass, succeeding where other infrastructure management tools both open and closed source have failed.

Also API upgrades shouldn't be a problem, even if you were using beta APIs. They are only really troublesome if you decided to indulge yourself with some alpha APIs before they were fully baked. If you don't do that then you won't run into any problems.

[busterarm]

Thanks, Mr. Well Ackshually,

First of all, those things you're talking about are installables in K8s. They don't come by default. Many people with (managed) kubernetes installations aren't even using them and get by just fine. Having them certainly isn't free and work certainly had to be done to build those for Kubernetes and likely are (or will be) trivial to implement elsewhere. I certainly was able to automate my certificate management infrastructure before I had Kubernetes.

The reality is that there are many companies out there that want like 10% of the features of an orchestrator like Kubernetes and don't need all of those features that make you think that it's a zero-sum game that Kubernetes has won.

The reality is that there are competing offers like Nomad that more-easily accomplish our technology goals and thus more attractive to enterprises with big budgets like mine.

Kuberenetes having the "critical-mass" that you speak of isn't to the exclusion of other competing tools having "critical-mass". Just like Oracle and DB2 aren't the dominant RDBMS of today.

As for calling people indulgent for using beta APIs, let's not forget that Deployment, StatefulSet, DaemonSet, ReplicaSet, NetworkPolicy and PodSecurityPolicy all started as beta APIs and were only removed in 1.16. And I imagine you wanted to use ingresses, but the old ingress.class annotation was deprecated and replaced with IngressClass in 1.18 and that was firm cutover for _everyone_. I can't imagine Kubernetes being very useful to many people without any of these...

[jpgvm]

There was no "Well Ackshually" in my reply.

I just pointed out that for everything you would need to build custom on Nomad there exists off-the-shelf components that will plug right into k8s. There is no way this would save money or development time, which was my main contention with your assertions.

You basically mischaracterized everything I said and then missed the entire point.

I don't disagree that there are some cases where Nomad would be superior. Off the top of my head if I wanted to build a modern rendering farm and I knew I wouldn't want to use the cluster for anything else then I would consider it over the HPC toolkits I have previously used for that (RIP Grid Engine) or Mesos which used to be king of that space.

The problem is these are incredibly niche cases and for 99.99% of companies they are better off swimming with the flow.

This is what I mean by "k8s has won". It's not that other things won't continue to exist and be created but most of them won't survive unless they commercialise within a niche or find a sufficiently large user that is willing to do the vast majority of the development (eg. Netflix Titan).

It's not a zero sum game but it's damn close. See here the corpses of Docker Swarm, Convox, Flynn ( :( ), original Deis, arguably ECS (it's a zombie at this point let's be honest).

The tide really turned when Azure and AWS were basically forced into offering managed k8s. In AWS case they also had to made deep modifications to VPC and IAM to properly support it. These investments wouldn't have been made unless forced which lends credence to the weight k8s has in the ecosystem. (Worth mentioning there is no hosted Nomad I'm aware of, their enterprise offering is install + support)

k8s is becoming the POSIX of distributed scheduling. The API you use to run diverse workloads over large numbers of machines that is relatively portable between companies. Right now there is still some vendor nonsense going on but over time it will be smoothed out.

The vast majority of distributed applications are going to be built targeting k8s as their runtime API. We can already see this happening but it will only increase over time.

To summarize I think it's fair to say things like "Nomad can sometimes be good if conditions x/y/z are met" but I think it's very dishonest to consider it as a viable competitor to k8s in the general case for most users because of the reasons outlined above.

[darkwater]

Thanks for this, I was really starting to think I was the madman.