[throwanem]

Yeah, I was doing "DevOps" back when it was still called "wow, good sysadmins are really expensive! I know..."

The rebranding is cute and all, and the tooling's a fair bit better, but nothing I've seen suggests that combining these two full-time roles in the same people is any more a good idea today than it was fifteen years ago.

[knightofmars]

Thanks for the flashbacks of supporting failing services at 3:00 AM. :D Even today, I am amazed at how absolutely terrible this type of decision making was and still is. The idea that you can "save some money" by having your engineers supporting your infrastructure "part time" is always going to cost your more in the long run. Because of the long-term effect of having an unskilled (in the area of question) individual attempting to do the job of a system admin (bad security practices, etc) and the turn-over you end up with due to burn-out.

[karmajunkie]

Whenever I've been in this position, or run teams this way, its been less about saving money and more about making sure developers have to live with their own bad calls on tech. I've worked in places where dev teams would just throw code over a wall with the equivalent of a sticker saying "Works on my machine!" and weren't the ones getting woken up in the middle of the night when that overeager query taps out the memory limits, and its a recipe for headbanging.

All things being equal (i.e. I've got the money to have someone who's role is more ops than dev but still does both) having that person to "own" the production configuration is valuable, but developers still need to be in touch with what their code does in production. Otherwise you eventually end up with the equivalent of a cool interaction design that's damn near impossible to implement on the web (another pet peeve of mine...)

[knightofmars]

There's a fundamental difference between holding engineers accountable for the downstream impact of their technical decisions and making engineers take on the added responsibilities of an additional position without compensating them accordingly for the added responsibility. In an organization that prioritizes stability there is an appropriate balance between engineering and system administration as well as potential overlap given the right boundaries and understanding of job responsibilities. The engineering team will be held accountable by the system administration team and changes will happen because of it.

The inherent failing in this structure is when one of two things happens; one, the system administration team does not have the appropriate channels (and clout) to provide push back against the engineering team; two, the technical teams (both engineering and system administration) don't have the ability to get technical debt payed off properly due to an improperly structured project management process.

Anecdotally, I've been witness to the second issue a number of times. If there isn't an immediate understanding of ROI for a proposed change then it isn't prioritized to be worked on. The thought process is generally along the lines of, "Engineers are an expensive resource, having them working on something that won't make the company money is obviously not the priority."

While some of this is on the engineering leadership as their job is to provide insight into ROI for technical matters there also needs to be a balance where the non-technical leadership trusts the technical leadership to know when to prioritize projects with non-obvious ROI.

[karmajunkie]

Definitely agree with much of that, though I don't think it detracts too much from my main point. Most of the companies I work with don't have the luxury of dedicated ops people, and those that do I still think insulate developers too much from the production environment.

[OpenDrapery]

It feels like QA is an antiquated notion these days. Why is a prod-like test environment too much to ask for? As a developer, if you call me in the middle of the night and tell me we are in crisis mode, my first thought might be "Did anyone test this thing with any kind of rigor at all?"

[karmajunkie]

i blame monolithic microservices...

[_jal]

I've been a sysadmin, I've been a software engineer, and now they call what I do devops. My experience is that the title is useless in terms of identifying what people actually do or how companies actually use them.

Some places just rebrand administration as devops. Shiny new title for you, stakeholders can be told we're not missing the trendy new hotness, carry on.

Some places will pull in a "devops person" and drop them on the IT team. This seems to usually result in something between "Jenkins? The devops guy sites over there" and incremental improvements.

Better than that is the model where the "devops person" is something of a liaison as well as an individual contributor - they sit in engineering standups, collaborate on how things are going to be built with an eye towards deployment and manageability, and generally play both sides of the fence. This, however, requires a "devops person" who is also a bit of a politician, and that's not usually contemplated when hiring. It also requires engineering to let the person meaningfully take part, which can become even more political some places.

Best, in theory, is more of less of a merger, but that does ignore that IT is a very different beast than engineering, and that the workplace has been dealing with the distinction for a very long time. Changing that is not a technical issue, it is a (big) management and perception issue. If you get it wrong, the power balance gets wonky and causes the usual problems when you have managers who are IT-clueless managing IT people[1].

All that said, and putting aside org-chart and political issues, I like to think two things are true:

- That someone who has done both engineering and administration is more likely to spot friction, second-order effects, and various traps and dead ends than someone who hasn't.

- That treating the various "devopsy" things - deployment, integration, monitoring, etc. as first-class requirements alongside product features leads to better results along multiple dimensions.

So it reduces to people-problems. There just aren't many folks who have lived on both sides of the fence who also want to do this sort of role[2]. And if you adopt "devops" as a cost-savings measure, you're doomed up-front.

[1] You never see IT managers promoted over engineering, for obvious reasons. The reasons why the reverse happens are obvious to me in descriptive terms, but not in normative terms.

[2] Of course, keep in mind that I'm in this category. I don't write this as ego fluffing - I believe these things to be true - but here's my disclaimer.

[eeZah7Ux]

Most of the modern devops hipster tooling is far worse and more complex to use that the old bash scripts put together by grumpy sysadmins.

[throwanem]

I've used those scripts - I've written those scripts! - and I have no idea what you're talking about. Puppet is a marvel, and when it's too heavyweight for the task at hand, there's Ansible. Both are far, far preferable to

    me@homebox $ for host in foo bar baz; do cat random_stuff_who_knows_if_it_works.sh | ssh $host 'bash -'; done

like we did in the old days you're talking about.

[marcosdumay]

Immutable configuration (puppet) is a godsend. No more digging into backups of etc and wondering who wrote something there, when (was it before or after the emergency?) and why. Now you place everything into a repository with commit logs like any piece of code.

That said, immutable configuration is static, so the need for grumpy admin style scripts did not go away. It just reduced.

[etatoby]

You can have my cryptic Bash one-liners when you pry them from my cold, dead hands.

But yeah, I agree that Ansible is the minimum that should be in place.

[throwanem]

Oh, I still perpetrate cryptic Bash one-liners myself, on a more or less daily basis. What's nice today, by comparison with the before time, is having a realistic option to do otherwise, and having that option not be Perl.

[marcosdumay]

> and having that option not be Perl

Oh, Python is quite old already :)

[throwanem]

Not better, though. Just different.

[eeZah7Ux]

Old bash scripts != cryptic one liners.

I had used very readable and documented scripts that would build images, boot VMs, deploy on physicals, configure network devices with hundreds of SLOC by simply calling the right CLI tools instead of (re)writing many thousands of lines in $language.

I was in a company around 2002 doing immutable deployments by updating images from git-tracked config files for whole datacenters.

E.g. calling to debootstrap takes few lines. Generating an ISO. Configuring PXE to serve it...

As a funny non-example: https://github.com/p8952/bocker

[throwanem]

That must've been amazing, back in the day! The organization was clearly far, far ahead of the curve, not least in that they adopted Git three years or more prior to its initial public release. Fantastic as it no doubt was, though, I'm glad that such awe-inspiring wizardry is no longer required to achieve a similar quality of result.

[rch]

I think originally 'DevOps' was intended to mean that dev/eng teams deserve to have a stable environment on par with c-suite email. Then it morphed into yet another way to pay people less by over-specializing them into tool operators.

[lmm]

The skillsets and tools have converged. Fifteen years ago sysadmins used shell, perl, and maybe little bits of C, while application developers used C++ or Java; these days both use Python or Rust or what have you.