[tayo42]

> I've found that most installs of k8s have been made using defaults, using tooling that Google has provided. We hired one such administrator, but when asked anything outside of how to run kubectl, they just shrugged and said "it never came up".

What is up with this? The last time I tried to learn kubernetes I couldn't find any information about how to set it up. Just some set up tools from google. I guess it is still like this? Is there really no one running kubernetes infrastructure with config management or anything?

[swozey]

The post you're replying to is absolute hyperbole. If you're hiring k8s guys who don't know etcd and the backend of k8s (we're not going to understand every single gear, I constantly forget how k8s garbage collects, I never have to interact with it) then you're not hiring Seniors who have worked on k8s for several years. That's no different from hiring a linux admin who only knows how to fix Cpanel. You made a bad hire or your budget wasn't high enough to attract experienced talent.

I'm one of the most frequent commenters on #kubernetes-users so I'm very aware of the questions and issues that come in from new k8s users and I'd say an absolutely massive majority of the users are running in baremetal via kubeadm/kops/etc. Typically on AWS (NOT EKS). The #gke channel is literally 1/10th the size of the #kubernetes-users channel.

If you have questions about k8s post in #kubernetes-users. The community is extremely helpful.

A LOT of people deploy K8s clusters via Terraform/Ansible, as well.

Why are professionals who know k8s back and forth less common? 2 years ago k8s was 1.1 and we had no idea where the market was going and if it would take off like it did. It takes time to build up the community and expertise. There are a LOT of very experienced k8s users nowadays whereas there were not 2 years ago. Finding someone with 2+ years of k8s experience who isn't a Xoogler is fairly rare right now because 2 years ago it wasn't the market behemoth that it is right now. I don't work with Google but I just happened to get involved with k8s almost 3 years ago. We are out there.

If you can't find an answer ping me @mikej and I'll try to get you going in the right direction.

[falcolas]

You call it hyperbole, yet you just also verified that the scarcity is a real problem.

> If you're hiring k8s guys who don't know etcd and the backend of k8s [...] then you're not hiring Seniors who have worked on k8s for several years.

> Finding someone with 2+ years of k8s experience who isn't a Xoogler is fairly rare right now because 2 years ago it wasn't the market behemoth that it is right now.

Indeed, there's not enough people who know how to run it for as broadly as it's spread; for how much it's hyped.

If there's a thousand people out there who have that level of experience, I'd be surprised. And in an industry running hundreds of thousands of clusters (or more!), that's just too few people.

[dsnuh]

Yeah, I'm not buying that there are thousands of k8s "Seniors" out there, and they just aren't the people that are being hired. I've been in operations for 20 years, and I think I would qualify for what you would call "good devops" in that I am a generalist with a wide breadth of experience in systems as well as programming. Kubernetes is a beast. We run it from scratch, in production, and just keeping up with the changes since the project inception can be maddening.

I did much of the early research POCs for my company when the idea of containerization really took off, and my deployments would seem to not have a shelf life of more than a few days before I would have to conform to some new method they came up with. I was using Tectonic when it was first released and the documentation would change underneath me as I would try to set up the clusters. It's a LOT to keep up with.

I can understand and explain to someone every protocol or idea underlying Kubernetes, sure, because they build upon standards that we have all used before in operations. But to try to understand how it is all working together within Kubernetes, and then add in the complex interplay if you are like us and integrate with non-k8s systems that have comlex firewall and routing rules now to allow the intercommunication... add in Calico or Flannel...Docker under the sheets with all its warts...it's a lot to manage. You need people that are engaged with the k8s project at a level that would normally be reserved for Googlers working on it.

Don't get me wrong... I like Kubernetes for the most part. I do agree that if you are planning to run in-house, you are in for some challenges, and that you will need a very high caliber of operations team to deploy and maintain it.

[swozey]

There are almost 24k people in the k8s.slack.io #kubernetes-users channel and it's only a small portion of the actual community. For instance I rarely see people from other huge K8s consumers, like Lyft, Zalando, Walmart, etc in the channel (high probability they just don't mention where they are or I never noticed).

I won't deny actual k8s experts are low in abundance right now. It's a complex platform in its near infancy. There are people brand new to k8s embarking on the journey to learn it every day in slack. Give them 6mo+ or another year and you'll have a few near experts and a ton of just generally experienced admins.

I don't feel as though it's any different from when I was working on my CCIE. When I was working on that there were only 18k other people out there who had CCIEs. I very rarely met one or even someone with just that level of skill (I'm not in the Bay Area/NYC). I had to ask questions through newsgroups and IRC and in IRC there were maybe 4-10 people at that level out of thousands. You could say there are thousands of networks out there that need a CCIE to run them but that isn't ever what happened; you'd have a CCIE basically lead from the top and their skill/experience would trickle down to lesser experienced netengs or they'd be brought in as consultants when necessary. I've worked with very few CCIEs. I see k8s going the same way. Every State will likely have a handful of experts on the subject while there will remain a ton of CCNA/CCNP level k8s admins and you just need to determine how complex your k8s infrastructure is and what level you'll need to hire to effectively manage it.

Brendan Burns goal is to democratize ephemeral infrastructure so that anyone who can code can manage it. That's another topic entirely but the community is starting to output enough general guidance in the form of blogs, books, slack, et al. that hopping into the ecosystem now is basically a breeze compared to what it was when I got involved in the 1.x days.

Could just be that I'm a masochist and like learning painful things.

My biggest complaint about k8s right now is the lack of real-world production knowledge being distributed. A lot of people set up a cluster and leave it and never optimize it or make it actually production-ready. My goal is to significantly accelerate that through training, blogs, etc.

[user5994461]

18k CCIE is magnitudes more than the current amount of kubernetes experts.

It's also not comparable. A network is setup only once by a CCIE. It's done and it doesn't need touching for the next 10 years.

A kubernetes cluster is setup once and then the troubles begin. Constant care needed every week for 10 years.

[swozey]

The first real CCIE certificate awarded was in 1993, #1025. 18k was something in like the year 2005 when I was in my early 20s.

Kubernetes was barely a blip on the map until 2017. There was Cisco routing and switching hardware LONG before there was even a CCIE certification.

This is a new ecosystem. Of course it takes time to develop experts.

A CCIE not touching a network for 10 years? That's ludicrous. I was a network engineer. I wish that was even remotely the case. I was constantly fighting firmware/bugs/etc. In fact I swore off Cisco and began working on my JNCIE at some point to stick with Juniper which of course had it's own issues.

I've run k8s 1.5 in production for 2 years on various clusters. That is almost a 2 year old release. I've had zero k8s specific problems. I recently migrated those clusters to 1.9 and apart from updating some API endpoints that changed over the major releases, and a lot of annotations that changed, it was very little actual work. It was mostly tedious "find & replace" work.

I'm not going to bullshit people. K8s has its bugs, quirks and is complex, but there seem to be a huge number of people who run away in fear on HN.

It can be understood. I didn't even know how to use Docker before I jumped into learning k8s.