[23david]

Remember, if you have a DevOps Team, then you are absolutely not doing DevOps. Developers should be involved in managing everything about their applications, including infrastructure.

Any sufficiently complicated infrastructure that has uptime requirements and significant revenue associated with it is going to have a DevOps Team (or the equivalent) ultimately responsible for ensuring that things are working. I guess it's possible to turn your entire dev team into part-time DevOps engineers, while still calling them Software Engineers, but I've usually found that doesn't work long-term and causes employee retention issues. It's like saying your company does 'No-Support' because you don't hire Support Engineers, while in fact you've enlisted your Software Engineering team to handle all support requests.

Also, if you're working in a regulated field like Healthcare or Finance, or anything that touches PII, your developers often can't have access to deploy code directly to production. Again, you could maybe work around this in the short-term by turning all developers into developers+devops, but they're different skillsets.

[dkersten]

Developers should be involved in managing everything about their applications, including infrastructure.

As a software engineer, there is just no way I can stay up to date with my craft as well as everything there is to know about infrastructure AND the all important security. I don't know about all the latest kernel patches and firewall rules. I don't know about monitoring failed hard drives. Etc.

I mean, I do know a lot about deploying and maintaining my running software, but Ops has a ton of other concerns that themselves are a full time job and I already have my time fully consumed by development tasks. I'm also ok with being pushed enough of the work so that I don't accidentally make software that's a PITA to deploy or manage and that I don't needlessly wake somebody else. That doesn't mean that I can take on the entire job (well, I can, but I wouldn't be doing as good a job, especially when security and such is concerned, and I wouldn't be doing as good a development job either). What next - DevOpsSalesMarkettingAccounting?

Much better, IMO, is to have small tightly-knit cross-functional teams which include an ops engineer.

[mnm1]

I couldn't agree more. Not only that, but trying to context switch between ops and development is a nightmare. The two positions are nothing alike. Most of my time as a sysdamin, I spent waiting and reading HN while super slow shit happens. It's basically a glorified secretarial position that deals with machines rather than humans. Every few minutes, I have to run another command, check output, respond to some AWS rep, etc. Basically, the absolute opposite mindset and skillset of what's needed to do proper development. After a few hours of that, switching back to development is almost impossible.

[0xdeadbeefbabe]

> well, I can, but I wouldn't be doing as good a job

You do a better job communicating with yourself though?

[dkersten]

Sure, but any advantage from that is lost in the fact that I've lost a ton of development time on doing ops tasks, which I don't have time to do properly and don't have time to learn how to do properly. Even bad communication to a competent ops person who does have time to, you know, do their job, and does have time to stay up to date, will result in much better results than if I tried to fit the work into my schedule.

For example, in my current work, I could try to figure all the ops stuff out, but I've already got a backlog of months of development tasks for a project that needs to be done asap.

[0xdeadbeefbabe]

I'd like to argue the advantage is sometimes lost, but not often.

[dkersten]

In my personal experience, writing complex software with complex infrastructure requirements, I have never had an advantage in not allowing other, more specialised people, to handle concerns outside my core competency (which is software developnent). Other people may be smarter or more efficient than I, or have more time available, or have simpler requirements.

[gloverkcn]

The statement touches on a real issue. I would revise the statement to:

If your DevOps people reside in a separate group outside of the development team, then you are not doing DevOps.

Architecture and Infrastructure go hand in hand, and DevOps is the glue that merges them together. I've worked with both DevOps and Architecutre groups in larger companies where Development, DevOps, and Infrastructure have been separated and siloed. This invariably leads to waste that undercuts any advantage provided by modern development practices.

The core of the issue is the designs will only consider trade offs within their skill set. Problems that are easily solved with a combination of dev/devOps/inf are instead solved using complex designs within a single skill set. The other end of the spectrum is that decisions get made in one group that will undercut the efforts of another group.

Real life examples:

The dev group wants to deploy backend service version 2 with fail back option in production. The API's are identical, the database is unchanged. Instead of using a load balancer and monitoring to automate role back the dev group builds another system with the job of routing traffic between the two systems, identifying failures, and then stop sending traffic to the new system.

The dev group designs a system with "micro-services" in mind. The developers have all containerized the services and run them on their laptop. The infrastructure and/or devops group doesn't want to deal with containers and instead deploys one service per VM. (In these scenarios the dev group will get a bill for the extra services).

The problem is that it's easier to grab the people sitting next to design something than scheduling a meeting with groups you rarely see. This is a key driver of Conway's law.

[Jare]

> If your DevOps people reside in a separate group outside of the development team, then you are not doing DevOps

Devops and swdevs sitting and working together, understanding each other's problems and needs, and keeping them in mind when designing solutions, is indeed a beautiful thing to watch.

[Lazare]

I think this is basically just an argument about terms.

If you write code, you're a developer. If you run servers, your in ops. If you are doing both, it's devops. So when discussing the idea of having a total separate team for running the servers, OP says it's not devops and he's right, because the definition of devops is not having the teams be separate. And you say that it's often totally needed to have that separation and you're also right.

> I guess it's possible to turn your entire dev team into part-time DevOps engineers,

I think you mean, it's possible to turn your entire dev team into part-time ops engineers. Which is 1) the entire concept of "devops", and 2) often a horrible idea. :)

(Sorry, but people who use "devops" to mean "ops" is a pet peeve of mine. If you're not a developer, it's just ops.)

[TheCoelacanth]

Most devops teams that I see don't manage the servers. They provide tooling that makes it easier for dev teams to manage their own servers.

If a devops teams is actually managing all the servers, I agree that they aren't doing devops, but at a large enough company it makes sense to have a team working just on things that make it easier for the company to do devops.

[throwanem]

Yeah, I was doing "DevOps" back when it was still called "wow, good sysadmins are really expensive! I know..."

The rebranding is cute and all, and the tooling's a fair bit better, but nothing I've seen suggests that combining these two full-time roles in the same people is any more a good idea today than it was fifteen years ago.

[knightofmars]

Thanks for the flashbacks of supporting failing services at 3:00 AM. :D Even today, I am amazed at how absolutely terrible this type of decision making was and still is. The idea that you can "save some money" by having your engineers supporting your infrastructure "part time" is always going to cost your more in the long run. Because of the long-term effect of having an unskilled (in the area of question) individual attempting to do the job of a system admin (bad security practices, etc) and the turn-over you end up with due to burn-out.

[karmajunkie]

Whenever I've been in this position, or run teams this way, its been less about saving money and more about making sure developers have to live with their own bad calls on tech. I've worked in places where dev teams would just throw code over a wall with the equivalent of a sticker saying "Works on my machine!" and weren't the ones getting woken up in the middle of the night when that overeager query taps out the memory limits, and its a recipe for headbanging.

All things being equal (i.e. I've got the money to have someone who's role is more ops than dev but still does both) having that person to "own" the production configuration is valuable, but developers still need to be in touch with what their code does in production. Otherwise you eventually end up with the equivalent of a cool interaction design that's damn near impossible to implement on the web (another pet peeve of mine...)

[knightofmars]

There's a fundamental difference between holding engineers accountable for the downstream impact of their technical decisions and making engineers take on the added responsibilities of an additional position without compensating them accordingly for the added responsibility. In an organization that prioritizes stability there is an appropriate balance between engineering and system administration as well as potential overlap given the right boundaries and understanding of job responsibilities. The engineering team will be held accountable by the system administration team and changes will happen because of it.

The inherent failing in this structure is when one of two things happens; one, the system administration team does not have the appropriate channels (and clout) to provide push back against the engineering team; two, the technical teams (both engineering and system administration) don't have the ability to get technical debt payed off properly due to an improperly structured project management process.

Anecdotally, I've been witness to the second issue a number of times. If there isn't an immediate understanding of ROI for a proposed change then it isn't prioritized to be worked on. The thought process is generally along the lines of, "Engineers are an expensive resource, having them working on something that won't make the company money is obviously not the priority."

While some of this is on the engineering leadership as their job is to provide insight into ROI for technical matters there also needs to be a balance where the non-technical leadership trusts the technical leadership to know when to prioritize projects with non-obvious ROI.

[karmajunkie]

Definitely agree with much of that, though I don't think it detracts too much from my main point. Most of the companies I work with don't have the luxury of dedicated ops people, and those that do I still think insulate developers too much from the production environment.

[OpenDrapery]

It feels like QA is an antiquated notion these days. Why is a prod-like test environment too much to ask for? As a developer, if you call me in the middle of the night and tell me we are in crisis mode, my first thought might be "Did anyone test this thing with any kind of rigor at all?"

[karmajunkie]

i blame monolithic microservices...

[_jal]

I've been a sysadmin, I've been a software engineer, and now they call what I do devops. My experience is that the title is useless in terms of identifying what people actually do or how companies actually use them.

Some places just rebrand administration as devops. Shiny new title for you, stakeholders can be told we're not missing the trendy new hotness, carry on.

Some places will pull in a "devops person" and drop them on the IT team. This seems to usually result in something between "Jenkins? The devops guy sites over there" and incremental improvements.

Better than that is the model where the "devops person" is something of a liaison as well as an individual contributor - they sit in engineering standups, collaborate on how things are going to be built with an eye towards deployment and manageability, and generally play both sides of the fence. This, however, requires a "devops person" who is also a bit of a politician, and that's not usually contemplated when hiring. It also requires engineering to let the person meaningfully take part, which can become even more political some places.

Best, in theory, is more of less of a merger, but that does ignore that IT is a very different beast than engineering, and that the workplace has been dealing with the distinction for a very long time. Changing that is not a technical issue, it is a (big) management and perception issue. If you get it wrong, the power balance gets wonky and causes the usual problems when you have managers who are IT-clueless managing IT people[1].

All that said, and putting aside org-chart and political issues, I like to think two things are true:

- That someone who has done both engineering and administration is more likely to spot friction, second-order effects, and various traps and dead ends than someone who hasn't.

- That treating the various "devopsy" things - deployment, integration, monitoring, etc. as first-class requirements alongside product features leads to better results along multiple dimensions.

So it reduces to people-problems. There just aren't many folks who have lived on both sides of the fence who also want to do this sort of role[2]. And if you adopt "devops" as a cost-savings measure, you're doomed up-front.

[1] You never see IT managers promoted over engineering, for obvious reasons. The reasons why the reverse happens are obvious to me in descriptive terms, but not in normative terms.

[2] Of course, keep in mind that I'm in this category. I don't write this as ego fluffing - I believe these things to be true - but here's my disclaimer.

[eeZah7Ux]

Most of the modern devops hipster tooling is far worse and more complex to use that the old bash scripts put together by grumpy sysadmins.

[throwanem]

I've used those scripts - I've written those scripts! - and I have no idea what you're talking about. Puppet is a marvel, and when it's too heavyweight for the task at hand, there's Ansible. Both are far, far preferable to

    me@homebox $ for host in foo bar baz; do cat random_stuff_who_knows_if_it_works.sh | ssh $host 'bash -'; done

like we did in the old days you're talking about.

[marcosdumay]

Immutable configuration (puppet) is a godsend. No more digging into backups of etc and wondering who wrote something there, when (was it before or after the emergency?) and why. Now you place everything into a repository with commit logs like any piece of code.

That said, immutable configuration is static, so the need for grumpy admin style scripts did not go away. It just reduced.

[etatoby]

You can have my cryptic Bash one-liners when you pry them from my cold, dead hands.

But yeah, I agree that Ansible is the minimum that should be in place.

[throwanem]

Oh, I still perpetrate cryptic Bash one-liners myself, on a more or less daily basis. What's nice today, by comparison with the before time, is having a realistic option to do otherwise, and having that option not be Perl.

[marcosdumay]

> and having that option not be Perl

Oh, Python is quite old already :)

[eeZah7Ux]

Old bash scripts != cryptic one liners.

I had used very readable and documented scripts that would build images, boot VMs, deploy on physicals, configure network devices with hundreds of SLOC by simply calling the right CLI tools instead of (re)writing many thousands of lines in $language.

I was in a company around 2002 doing immutable deployments by updating images from git-tracked config files for whole datacenters.

E.g. calling to debootstrap takes few lines. Generating an ISO. Configuring PXE to serve it...

As a funny non-example: https://github.com/p8952/bocker

[throwanem]

That must've been amazing, back in the day! The organization was clearly far, far ahead of the curve, not least in that they adopted Git three years or more prior to its initial public release. Fantastic as it no doubt was, though, I'm glad that such awe-inspiring wizardry is no longer required to achieve a similar quality of result.

[rch]

I think originally 'DevOps' was intended to mean that dev/eng teams deserve to have a stable environment on par with c-suite email. Then it morphed into yet another way to pay people less by over-specializing them into tool operators.

[lmm]

The skillsets and tools have converged. Fifteen years ago sysadmins used shell, perl, and maybe little bits of C, while application developers used C++ or Java; these days both use Python or Rust or what have you.

[heisenbit]

Remember, if someone claims something absolutely then there is absolutely the chance that this gentleman is partially wrong.

[tdb7893]

Or as I like to quote at work "only a Sith deals in absolutes"

[pacala]

+1. Nothing promotes the ability to produce quality code like being woken at 3am to troubleshot a problem 3 microservices away from your own.

In my anecdotical experience, the ability to recover from such incidents degrades with age, so nowadays I regard "everybody [but management] does DevOps" as codeword for ageism.

[solatic]

> Any sufficiently complicated infrastructure that has uptime requirements and significant revenue associated with it is going to have a DevOps Team (or the equivalent) ultimately responsible for ensuring that things are working.

I don't think these are mutually exclusive; I think you can both have a dedicated applicative operations (rebranded DevOps or SRE) team and still allow developers to rapidly provision and deploy. The point is a change in responsibilities and workflow on the part of operations: whereas their task was originally to deploy, verify success, rollback if necessary, monitoring, etc., their new task is to embed those tasks in code, so that developers can safely and reliably perform those tasks by themselves. You can have an organization where developers can automatically provision infrastructure which is behind your enterprise firewalls, hooked up to up-to-date enterprise artifact repositories, has enterprise PKI management already set up, etc., because expecting developers to know the ins and outs of setting that up manually is moronic and much of that should be scripted for ease of management anyway.

[enknamel]

> Remember, if you have a DevOps Team, then you are absolutely not doing DevOps. Developers should be involved in managing everything about their applications, including infrastructure.

I think the intent here is a good one. You as a developer should be involved with how your software is deployed and managed. It's just that there is a pretty grey boundary between where developer involvement ends and ops involvement begins.

Another issue is that devops is extremely poorly defined. Some people insist this means only writing chef scripts. Some people it's diagnosing and fixing production issues. Some people it's just sysadmin work. Or other responsibilities and combinations.

[pjc50]

Where's that quote about how software architecture evolves to reflect the organisational structure of the business?

Amazon does it this way because it's made of teams that are not just loosely coupled but somewhat firewalled from each other, along with a brutal internally competitive process.

[devonkim]

Conway's Law

[taneq]

Don't they have explicit rules dictating that any software must work as a service under failure-prone conditions?

[whateveracct]

More than that, internal Amazon tools make it stupidly easy to do everything-as-a-service, so it's just how everyone does it.

[dominotw]

I always thought DevOps was the guy who is writing Kubernetes, not the guy who is installing it. Not sure why a special title is needed though.

[eeZah7Ux]

Dev + Ops == Devops, get over it.

[dang]

Please post civilly and substantively, or not at all.

[eeZah7Ux]

Sorry, I'll contextualize.

> is going to have a DevOps Team

The parent is arguing that the concept of a "DevOps Team" that does Ops makes sense and is not an oxymoron. Yet "DevOps" literally means doing Dev and Ops together.

Otherwise have a "Developer Team" and an "Operations|Engineering|SRE|SysAdmin Team".