All is fine and dandy and BSDs can solve many use cases. Unfortunately for the solution we are working on, which imply many microservices we need Kubernetes and no BSD equivalent to Kubernetes exists.
I've helped build two top 10 service provider networks (10s of Tbps). One on FreeBSD, and one on Linux with Kubernetes.
I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work.
But it also seems to come with a magnitude of accidental and ornamental complexity. I would imply the same about microservices versus, say, figuring out your repository, language, and deployment pipelines to provide a smooth developer and operator experience. Too much of this industry is fashion and navel gazing instead of thinking about the core problems and standing behind a methodology that works for the business. Unless google moves its own infrastructure to Kubernetes, then maybe there's something to be had that couldn't reasonably be done otherwise :)
We went from a virtualized server model to managed Kubernetes and costs have escalated considerably. The additional complexity and maintenance overheads of Kubernetes are not trivial and required additional staff hires just to keep things ticking. I think the cost so far from moving from two cages in separate datacentres running blades to AWS is approximately a 6x multiplier including staff. This was all driven on the back of "we must have microservices to scale", something we have failed entirely to do. It's a complete own goal.
I think it mainly works for startups which can build it as a greenfield and are extremely sensitive to scaling (either they are exploding within 2 years or they're dead within 2 years). So they plan to scale from 0 to millions quickly. For that kinda business it makes sense because once the exploding growth comes it's super easy to scale up and even do that automatically. That's the power of Kubernetes.
For an existing business not in a hyper-growth phase with lots of restrictions and policies and processes and legacy stuff it's just extra work with no gain. It's just a round peg in a square hole. Wrong tool for the job.
Kubernetes has unquestionably been a game changer.
If you look at most enterprises today you will see it deployed everywhere.
And most of the complexity has been abstracted away by the cloud providers so all you're left with is a system that can handle all manner of different applications and deployment scenarios in a consistent way.
> Kubernetes has unquestionably been a game changer.
> If you look at most enterprises today you will see it deployed everywhere.
This doesn't mean it's a game changer. It just means it has a big cargo-cult and people keep using it regardless of whether they need it.
> most of the complexity has been abstracted away by the cloud providers
The abstractions that people add make things more complex, not less. Unless of course you don't care to understand what you're running, which is precisely the problem with the culture around most folks that use kubernetes (and more broadly the US cloud providers as an entity).
Kubernetes is one of those things that are only as complicated as you want it to be.
The problem with Kubernetes and complexity is that because it simplifies a lot of things that are a huge PITA to accomplish on a "homemade" server/container setup is that there are a huge number of products and things you can run on kubernetes to "do stuff".
And it is hard for a lot of people and organizations to resist the "oh shiny" aspect. Stuff like "Oh, look I can do network policies and service meshes!" or "Lets create this really complicated and big thing so we can configure all our AWS infrastructure with kubernetes objects! Who cares that a bad commit can destroy the infrastructure the cluster depends on along with our ability to manage any of it!" or "Look we can have lots of namespaces for all our internal orgs and departments, lets make a gigantic centrally managed Kubernetes cluster that will be managed by IT and that everybody will be forced to use at the same time! Putting all our eggs in one basket is a awesome idea!".
K8s sorta removes the barrier of entry that world normally stop people from implementing those sorts of bad ideas.
Otherwise the core vanilla Kubernetes isn't really that complicated compared to most DIY solutions that try to manage large numbers of apps on clusters of systems. Most of the time it ends up a lot more robust and simpler in the right hands.
You're not wrong about the ideal case, but I've never seen the ideal case in any real (read: paid) context. I've seen hobbyists with very solid setups that were well managed, I've made the thing that kubernetes wants to solve by hand and I understand it's not a trivial (that is to say, not a single afternoon of design and implementation) to do.
I guess I'm just really frustrated with seeing the people that get paid to use and promote it be so bad at what they do and I don't like cleaning up after them :)
> The abstractions that people add make things more complex, not less
Everything in our industry is built on abstractions.
By your logic we shouldn't be using operating systems, compilers, libraries etc and writing our own custom software to manually move applications between different hosts if they go down.
I didn't say this at all. I said you should understand what you run, and if you understand it you realise it adds complexity.
I don't really see how this is a particularly controversial statement and I will say I don't appreciate you trying to reduce it to some ridiculous extreme. You didn't add anything to the discussion and it's tragic, because you could have made a nice point about system boundaries, levels of abstraction and perhaps information overload.
That would have made a great point about why abstraction is sometimes necessary. Then we could have spoken about what makes good and bad abstractions. Perhaps someone who came upon this thread later could have learned something and we'd have made the site a better place together.
You're probably going to be downvoted to oblivion but I 100% agree with you. It seems like they tried to remove all the complexity, and just made new complexity.
Modern systems are complex generally. If it's not Kubernetes with YAML files, it's a bunch of servers in VMware which is own ball of wax followed by extensive Ansible/Puppet/Chef setup and SREs who keeping entire architecture in their head since they don't have time to write it down.
Obviously, there is exact opposite with stuff like fly.io but they can be extremely constraining.
Yeah, I really liked Heroku and I used it for so many years, but it also killed itself kinda. It could never reduce prices as all machines "was the same" and thus a reduction in price meant lose of revenue and now they're just super expensive comparably.
Now I just dump Caddyserver and install things bare metals servers. I avoid all dependencies like the plague
You will be happy to learn that it was not in fact downvoted at all. I think this may be the first time I've expressed this sentiment to (mostly) positive feedback and I'm pretty sure it's because it was explained properly this time. That is I think a good thing - the downvotes that came before gave me the impulse to improve, and I was rewarded when I did.
Kubernetes is absolutely not an accidental complexity. Microservices are not an accidental complexity, and they are not a replacement for a proper repository.
Kubernetes solves administration of a cluster of Linux machines, as opposed to administering a single Linux machine. It abstracts away the concept of a machine, because it automates application scheduling, scaling across different machines, rolling updates of applications, adding/removing machines to the cluster all at the same time. There are no instruments like that for applications, the closest to them are something like Spark and Hadoop for data engineering tasks (not general applications).
Microservices are also used to solve a very specific problem – independent deployments of parts of the system. You can dance with your repository and your code directories as much as you want, if you're not in a very specific runtime (e.g. BEAM VM), you will not achieve independent deployments of parts of your service. The ability to "scale independently" (which tbh is mostly bullshit) is an accidental consequence of using HTTP RPC for microservice communication, which is also not the only way, but it allows reuse of the HTTP ecosystem.
FreeBSD has jails, which AFAIK are fully a match in terms of functionality, but you would need to finish implementing containerd support (maybe via runj?) and then get k8s to run on top of that. I seriously doubt there's any actual technical blockers, it's just that nobody's finished implementing all the pieces and putting them together.
I don't really see Kubernetes as being a game changer. The biggest pro, it makes it easier to onboard both development and operations personnel having a quasi-standard for how a lot of things like scheduling and application networking work.
But it also seems to come with a magnitude of accidental and ornamental complexity. I would imply the same about microservices versus, say, figuring out your repository, language, and deployment pipelines to provide a smooth developer and operator experience. Too much of this industry is fashion and navel gazing instead of thinking about the core problems and standing behind a methodology that works for the business. Unless google moves its own infrastructure to Kubernetes, then maybe there's something to be had that couldn't reasonably be done otherwise :)