[JohnStudio]

I think the entire statosphere of DevOps is just about dead on the whole in 2018 .. in retro, after working with things like Docker .. and more specific industry variations beyond the Amazon tech, it makes no sense to dwell on the security / control of a dedicated systems admin professional since the tools are all outside the local domain anyway. The rest from VoIP to IoT to container services are managed whole-sale .... SysAdmin is a dino in the age of distributed tech and outsourced IT resources.

I'm a programmer, so I'll take heat for it .. but I don't see a need for them anymore.

[nunez]

Nah, you’re right

The entire purpose of DevOps IMO was to close the gap between sysadmins and devs through code. Devs doing everything, including infrastructure, was and is the entire plan! Public cloud made this super duper easy.

The problem is devs don’t want to manage core infrastructure (VPCs, networking, modules for deploying lambdas and database clusters and container orchestration clusters, etc) and somebody has to do that stuff

Ideally, those would just be features like any other software team, as it’s all API calls at the end of the day. But lots of companies have issues with structuring their platform teams like software teams because its “not software” even though it is

This problem is more deeply entrenched at large companies with hundreds of millions of dollars of compute that they own that is owned by an old school IT function that can’t fathom the idea of either giving it up or making it accessible like cloud and would rather pay VMware tons for tools that make teams even slower than have their sysadmins become developers

Then there’s the whole protectionist “You’re taking my job” and “devs can’t possibly know this much about $infra” that isn’t dying off anytime soon

It’s complicated

[mmt]

> Ideally, those would just be features like any other software team, as it’s all API calls at the end of the day. But lots of companies have issues with structuring their platform teams like software teams because its “not software” even though it is

Just because it's is implemented as API calls at the end of the day doesn't necessarily make it not "not software" (if you'll pardon the double negative), at least in the sense that I believe you mean.

To whit, I believe you're suggesting that if something can be expressed as code, it's all "software" and can therefore be designed, written, and maintained by the same kinds of experts, software developers.

I disagree, because the nature of the infrastructure-as-code code is too different from the application software code.

One could, similarly, express an FPGA configuration in code, but a software developer would not automatically be good at programming one. This is even likely to be true for less extreme examples, such as programming expertise not automatically transferring from general software (for lack of a betterm term) to code that works well on, say, GPUs.

In the case of IAC "software", a more mature design is more likely to resemble traditional sysadmin/network/security best practices than application software features. It could also have significant financial side effects if there's an error, assuming public cloud, which could require more stringent standards of control, review, and quality, especially if a company ends up in SOX territory.

>Then there’s the whole protectionist “You’re taking my job” and “devs can’t possibly know this much about $infra” that isn’t dying off anytime soon

I'm sure some of this exists, but my own experience is an attitude not that devs can't know a certain amount about infrastructure but that they simply don't, often because they actually don't want to.

Perhaps they fear that if they do end up knowing that much, they'll end up being the ones to manage that core infrastructure, which you indentified that they don't want to do!

[mmt]

I'm not convinced there was ever a need for sysadmins, especially from the point of view of programmers. Programmers have always been able to do that work themselves.

It's just that they don't want to. They think it's boring, or perhaps even beneath them. It's a bit like tax preparation.

I don't think that's really changed all that much, even today, even if it's now programming against the AWS API, judging by the number of job postings for such a role.

Also, like with tax preparation, using a good expert instead of doing it oneself could save on a lot of money and/or future headaches, but it can come with "grumpy" old-fashioned-seeming nagging about procedures akin to annotating and saving receipts. Whether it's worth it depends on situation and, far more importantly temperament.

[tonyarkles]

> I'm not convinced there was ever a need for sysadmins, especially from the point of view of programmers. Programmers have always been able to do that work themselves.

> It's just that they don't want to. They think it's boring, or perhaps even beneath them. It's a bit like tax preparation.

I'll half agree here, and half not. I'm coming at this as a programmer who has historically done ops as well, currently doing more ops than dev.

There's parts of it that programmers don't want to manage, like Apache configuration, provisioning and sizing VMs appropriately, etc. That's the boring stuff, and I totally get it. I don't like managing most of that stuff either, and have got everything set up with service discovery to handle a good chunk of the configuration, and Terraform for managing resources instead of having to click through the Azure Portal, and Nomad for scheduling tasks in the clusters. Groovy.

There's also the parts that the developers seem to have no clue about. They're very happy with all of these abstractions, but there's a lot of layers underneath. They think in terms of making HTTP requests; when those don't work, it's up to me to figure out that their library is keeping a pool of HTTP connections open, and the Azure load balancer silently drops those connections out of the NAT table after 4 minutes of idle time (without sending FIN or RST packets to either side, natch). When the app gets terminated because it's exceeded its cgroup memory limit, I'm the one helping them adjust their JVM heap size. And when their requests take too long, I'm there helping them look through their query logs to figure out what's happening.

All of these cases have happened in recent memory, and it's never been a matter of "this is boring and/or beneath me", it's been a matter of "HALP! I HAVE NO IDEA WTF IS HAPPENING"

Edit: so yes, I agree with you in the sense that I'm a developer who generally speaking doesn't need a sysadmin. I installed Linux for the first time when I was 12 years old, 22 years ago :), and I didn't have a sysadmin then. In those 22 years, I've seen a lot of weird shit happen, and most of the people I've worked with don't seem to be well equipped to dive 3 abstraction layers deeper than they're used to to figure out what's wrong.

[mmt]

> currently doing more ops than dev

I'm not sure you're even half disagreeing with me here, especially as you just admitted you may have the same bias as I do :)

> All of these cases have happened in recent memory, and it's never been a matter of "this is boring and/or beneath me", it's been a matter of "HALP! I HAVE NO IDEA WTF IS HAPPENING"

These two cases strike me as actually one case, with the former being the cause and the latter being the effect. At least that's my contention: they had no idea wtf is happening in those lower layers because learning them was beneath them (no pun intended). Nothing was really stopping them from learning about heap size versus process/cgroup limits.

On the other hand, you do bring up an important point about (mis)behavior of network infrastructure. That's not something the average programmer would necessarily encounter or have access to, especially at scale, and therefore wouldn't be expected to know. Still, my point still stands: there's nothing stopping programmers from gaining that knowledge if/when it becomes necessary, should they want to.

Now, I'm not trying to play dumb. I mostly don't want to read too much between the lines. Are you, in essence, saying that your half-disagreement is that sysadmins are needed, not because (average) programmers don't want to learn all those layers, but, instead, are incapable of learning them?

[tonyarkles]

Very very excellent points. I think that everyone on my team would be capable of learning them, but at least in this specific case it's not so much that they feel it's beneath them, but more of a fear. They talk about C as if it's something that only dark wizards understand, likely as a result of poorly taught early CS.

I think what I'm saying is that I feel like sysadmins are frequently needed for sake of expediency, especially when hiring younger developers. And maybe, due to that same bias, I actually mean "senior developers who understand abstractions several layers deep" :)

[mmt]

> I think what I'm saying is that I feel like sysadmins are frequently needed for sake of expediency, especially when hiring younger developers.

Thanks. That's a point I hadn't considered.

I'm not sure that expediency translates to a need, as such, but that situation is certainly different from the one I envisioned (where a sysadmin is merely a luxury or an optimization to the programmer-DIY-ops scenario).

> I actually mean "senior developers who understand abstractions several layers deep" :)

I'm not sure you do, since you admitted to really being at least part sysadmin, earlier :)

[tonyarkles]

> I'm not sure you do, since you admitted to really being at least part sysadmin, earlier :)

Hah!

I wish I could remember which talk it was. Bryan Cantrill had a good line about DevOps in one of his talks (I think it was this one: https://www.youtube.com/watch?v=30jNsCVLpAE). The gist was "you can say you're DevOps, but when the shit hits the fan, you're either going to be Dev or Ops. If you're a Dev, you're going to want to debug the problem before rebooting the failed machine. If you're Ops, you're going to want to reboot the machine as fast as you can to get it back up." Through that lens, I'm definitely pretty far over on the Dev spectrum; when something goes catastrophically wrong and someone reboots a box to "solve the problem", my first reaction is "YOU FUCKER YOU BURNED THE CORPSE"

It's such a tricky thing all around. Looking back at what I wrote earlier, I also realize that in some ways I'm facilitating the ignorance. I've got a really nice Consul and Nomad setup for the team, so they can pretty much just toss .wars and Docker containers at the cluster and they'll automatically get scheduled somewhere with spare capacity. The load balancer, the database cluster, all of the service discovery and job scheduling stuff... they've never had to get in and set any of that up. Maybe it's time to do more mentoring...

Anyway, thanks for getting me thinking. I've been a little bit grouchy lately about all of the recent experiences of people not knowing how the stuff they build actually runs. You've been a great mirror for some self-reflection :)

Edit: also, bare metal rocks.