| <rant> And yet finding people who can reliably install K8s from scratch, who understand what's going on under the hood, remains remarkably close to 0. How many people can, within a few hours, tell you how Kubernetes runs DNS, and how it routes packets between containers by default? How do you run an integrated DNS which uses, say, my_service.service.my_namespace instead of my_service.my_namespace? I've found that most installs of k8s have been made using defaults, using tooling that Google has provided. We hired one such administrator, but when asked anything outside of how to run kubectl, they just shrugged and said "it never came up". The codebase is vast, complicated, and there are few experts who live outside of Google. And it's getting more vast, more complicated on a quarterly basis. It bothers me how far operations has gone from "providing reliable systems on which we run software" to "offload work onto the developer at any cost". </rant> I realize that a lot of this is because of scarcity. The good devops folks (i.e. those who are both competent generalist sysadmins and competent generalist programmers) are few and expensive. That makes pre-packaged "full stack" solutions like GAE, Kubernetes, and Fargate very appealing to leadership. "You don't need an operations department to act as a huge drain on your revenue, just re-use your developers" holds a lot of appeal for those high up in the food chain. It's even initially appealing to developers! But in the end, it makes as much sense as re-using your developers to do customer service. |
I've abandoned operations as a career path and have now gone into product management, but I was an operations person for more than 12 years. In that time frame I learned very quickly that upper management considered the operations teams to be "system janitors" and that developers considered operations engineers to be their inferiors. The "move fast and break things" attitude is great sometimes, except it gives license to shortsightedness.
The reality is that operations is not a specialized skillset, in fact it's a generalized skillset made up of being a specialist in multiple facets of complex systems. There's simply not that many people out there who have that level of knowledge and understanding, and the industry has both perpetuated this problem by treating operations people terribly and worked around this problem by focusing on building stacks that require minimal operational overhead. Any good operations person could have been a software developer, but wanted to get beneath the abstraction layers. Instead, we get treated worse, paid less, and have less job demand despite being more competent. Most of the best ops people I've worked with ended up either leaving ops entirely, like myself, or becoming software developers to get a pay bump.
Luckily I got to work for a few decent companies along the way in my career that treated me well and I made a lot of life-long friendships with very smart people as well. So don't read the above as some deep complaint. It's just an observation of the reality that the incentives aren't there for smart and talented people to invest their energy in operations. I advise most of the young people passed my way to become software developers. They'll have more autonomy, get paid more, have higher job demand, and get treated better in general.