[rubiquity]

> and not something that wasn't designed to create jobs for consultants.

I wouldn't be so sure about that. Consulting companies are drawn to operational complexity, of which Kubernetes has a lot of, like bears are to honey.

https://kubernetes.io/partners/

https://www.google.com/search?q=kubernetes+consulting&ie=utf...

[atombender]

Kubernetes came out of the engineering team at Google. It was created by engineers for engineers. The fact that it's being exploited by enterprise consulting types doesn't change that premise.

[rubiquity]

I like to believe that nobody creates anything for sole intentions of extracting consulting bucks (there probably are cases of this happening in tech, but I like to be benevolent). So I don't believe Kubernetes was created for that and I don't think that OpenStack was either. But regardless of what the creators want, opportunists see new platforms and want to attach themselves to them.

Kubernetes isn't simple. Some of that is due to the problem its trying to solve, some of it is because the design, some if it is because the ecosystem that has evolved around it with regard to meshes/load balancers/control planes and overlay networks. We are after all replying on an article about a startup ecosystem being created around Kubernetes.

Tools like Kubernetes gaining adoption are partly based on technical merit/need and part based on fashion. Developers like toys and for some people the toys are what gets them through the day. Most of the people adopting Kubernetes probably don't need it. Developer Advocates and consulting shops are the ones making people think they need Kubernetes when they don't. This is an industry problem and that's all well and good. Let's just not pretend there isn't some influence from startups (typically backed by VC, which exacerbates the pressure) and consulting shops.

[bonesss]

> I like to believe that nobody creates anything for sole intentions of extracting consulting bucks

Sharepoint and BizTalk spring to mind... MS Dynamics (or whatever), too.

SAP and other "configuration heavy" systems bring it to another level.

> Most of the people adopting Kubernetes probably don't need it

If they geniunely don't need it then it will bite the in the ass.

On the other side: the absolute nightmare hodge-podge of half-baked non-integrated solutions that Kubernetes has murdered, and the promises it delivers on for cloud-native and cloud-first systems, are major strategic wins with clear ROI...

MS does not profile Google tech without cause. The largest virtualization supplier does not integrate competing Google tech without cause. Amazon is an easier mark here, but: the very fact there is a common, turn-key, orchestration platform across all major cloud providers is massive, and very hard to attribute to developer advocacy.

[SmellyGeekBoy]

> MS Dynamics

Way to trigger my PTSD...

[daxfohl]

But there's this from a fairly influential former googler: https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48...

It's a quick read, but to summarise: it's almost as complex as google's internal borg system but the benefit isn't even close (partially because nobody else has google problems). I can't down K8s personally, as I've never used it myself. But I wonder if there's the possibility of a system that's more 80/20 of google's borg rather than the seeming 20/80 coming from k8s. And I wonder if Grab will release it next year.

[lobster_johnson]

I don't think that's true. If you believe the team that designed Kubernetes, Kubernetes is an attempt to improve on Borg.

Borg is an accumulation of a decade's work with containers at Google, and has been described by googlers as a rich but a little messy, having been designed incrementally over many years as needs have surfaced. Borg could never be open-sourced because it's so specific to Google; for example, it uses Google's own cgroups-based container tech, not Docker/OCI/etc. Omega, as I understand, was an effort to clean up Borg and modernizing it, but apparently it was never put into production; instead, some of the innovations ended up being backported to Borg [1].

More importantly, Kubernetes is based roughly on the same design as Borg: A declarative, consistent object store, with controllers, schedulers and other bits and pieces orchestrating changes to the store, mediated by a node-local controller (Borglet/Kubelet). A major difference between Borg and Kubernetes is that with Borg, the object store is exposed to clients, whereas Kubernetes hides it behind an API. Another difference is the structure of containers; Borg's "allocs" are coarser-grained than pods and Borg is less strict about where things go, which googlers have described as a shortcoming compared to Kubernetes' strict pod/container structure. Another difference, also seen as a shortcoming, is that Borg lacks Kubernetes' one-IP-per-pod system; all apps on Borg apparently share the host's network interface. Kubernetes also innovates on Borg in several ways; for example, Borg doesn't have labels [2].

Borg, from what I gather, scales much further than Kubernetes at this point, but it's really not related to the design. The design is fundamentally the same.

Yegge's criticisms are too handwavy ("overcomplicated") to counter, but I don't think Yegge knows what he's talking about here. As for "benefit": Not sure what you mean by this, but Kubernetes arguably comes with benefits — declarative ops, platform abstractions, container isolation — even if you're just running a single node. The notion that you only need Kubernetes if you have "Google-scale problems" is just nonsense.

PS. What's "Grab"?

[1] https://ai.google/research/pubs/pub44843 (I recommend reading this paper)

[2] https://kubernetes.io/blog/2015/04/borg-predecessor-to-kuber...

[pathseeker]

All of the things you described as improvements are more complexity and layers of indirection. Kubernetes may be an attempt to improve on Borg but adding on a bunch of features and plugin architectures to solve more use cases isn't necessarily an improvement.

[coredog64]

Is your assertion that k8s suffers from "second system syndrome" when compared to Borg?

[puzzle]

Kubernetes is actually the third system. In-between, there's Omega. Yegge must have never set up a GSLB or GFE service to appreciate the extra stuff that Kubernetes features.

[daxfohl]

Thanks for the detailed insight. I've been following k8s for a while now but never had a need, and Yegge's post put an end to it in my mind.

Grab is the company Yegge left google for. He always complained about google's inability to platformize, so random hunch is he instills this desire into Grab? But entirely random. I also don't know how influential he was inside google vs outside.

[vorpalhex]

Kubernetes does suffer from complexity right now, but that also gives it it's flexibility.

I think turnkey kubernetes solutions like Rancher will dominate for a lot of use cases, especially for individual devs and small teams that can't have a dedicated DevOps resource to manage kubernetes.

[sleepybrett]

As someone who has been running kubernetes clusters for the last 1.5-2 years. It's quickly becoming boring (in a good way). I'm at a fairly large enterprise and we're running five clusters (two gke managed, which .. is limiting, three with our own bootstrap). I'd say we spend about 10%-20% of our time actively managing clusters (upgrades, troubleshooting.. some of that time is about maintaining some value adds like automatic log shipping.. that kind of thing), most of our work is writing value-adds, helping customers onboard and consulting with teams on how to best deploy and work with kubernetes (many teams at my company aren't really familiar with containerization).

Granted we run our own bootstrap (when we started none of the bootstraps running around we're ready, we started with a terraform/make implementation of hightower's hardway and we've just kept adding. We're thinking about revisiting the space again.

[emodendroket]

I didn't know he was still writing.

[justicezyx]

- K8s is not a clone of Borg

[user5994461]

Every tech companies with thousands of employees has google problems.

[yongjik]

I highly doubt it. Even most of Google doesn't have "google problems". (Having to design things for "google scale" when you have no reason to was a popular gripe while I was there.)

You don't go Google scale because it's cool to. You either do it because you absolutely have to, or you don't because, thankfully, you don't have to.

[daxfohl]

It sure is trendy to think so though!

I'm currently supporting ops of some "eventually consistent, globally distributed" (if you can fathom through the previous engineer's algorithms) built-from-scratch system designed to be resilient in the face of multi-megaX-transactions-per-second, that currently contains all of 8K records.

Whereas stackoverflow.com runs on SQL. Their own hardware at that.

I half wonder if a large chunk of AWS's revenue could be replaced by https://www.amazon.co.uk/electrical-sockets-adhesive-sticker...

[user5994461]

Companies don't go google scale, companies are google scale. It's not a choice, it's a fact they have to deal with.

Pick any large tech company, they have thousands of servers and countless customized software running. That's what is meant by google scale.

They need to manage that and they suck at it most of the time, they don't know what resources they own and they can't figure out what's running or where.

[pathseeker]

>Kubernetes came out of the engineering team at Google.

Attaching the Google name to projects stopped having meaning ~5 years ago. Their hiring process optimizes for fresh graduates who know nothing about engineering. Something coming out of Google means nothing, especially when Google itself doesn't even use it!

>It was created by engineers for engineers.

So was openstack. Then the feature creep happened.

[lobster_johnson]

Google does use GCP/GKE internally. And Google's internal Kubernetes team is basically the GCP team, according to googlers on HN.

[sitkack]

Just because Google made something doesn't mean it has sound engineering behind it. Eng at Google is also rewarded for complexity.

[outworlder]

It has a lot of complexity, but only because it does so much.

Thing is, can you point to alternatives which are simpler? I cannot think of any, unless you only have a handful of containers to manage.

Even things like AWS's own ECS become difficult to manage as you grow.

[yebyen]

> only because it does so much

> can you point to alternatives which are simpler

Sure, there are plenty of simpler alternatives... as long as you're willing to consider ones that don't do quite as much. (Beware, you'll also need more of them.)

[bonesss]

Amen, sister.

The complaints that Kubernetes is "complex" are obviously comparing it to more targeted solutions and ignoring the aggregate system that arises in its place.

If you find a tool with a better fit to the problem, then yay for you do the smart and simple thing. In practice, though, for many shops there's a major hodge-podge of almost-there solutions which require much more specialization on much less popular frameworks with much less vendor support to compete with Kubernetes.

Kubernetes did not come into a vacuum: it's disrupted some mature solution ecosystems by being much, much, better.