[curun1r]

For what it's worth, I'm older than you and started in ops. If it helps to put things into context, I spent a ton of 1997 using a chisel to shave a bit off standard RAM to help avoid Sun's near 10x markup. The startup I worked at was provisioning around 20 Sun boxes per week and we had to patch together tools to manage infrastructure of that size before off-the-shelf tooling to do that really existed. I wrote a lot of expect in 97-98.

Where I get frustrated with ops is when they forget that they're in the service business. They're there to serve application developers in the same way that the application developers are there to serve customers. An application developer pushing back at a customer ask with the line, "without really stopping to ask what progress is from the perspective of developers" would be rightly lampooned. And yet somehow there's people that think the ops mindset is reasonable. In my view, "no" is not a valid viewpoint. Application developers are telling you they want the kind of functionality that Docker offers. That makes it your job to figure out how that happens, not if it happens.

[Karunamon]

Where I get frustrated with ops is when they forget that they're in the service business. They're there to serve application developers in the same way that the application developers are there to serve customers.

In my view, you've got that completely backwards. Both departments serve customers in a roundabout way. Dev is responsible for turning user stories (generated either by other devs, focus groups, and so forth - which may, but doesn't necessarily involve an actual paying customer) into code, ops is responsible for keeping that code running.

If dev screws up, user gets bugs, or the app goes down. Same for ops - it's just the class of bugs are different.

Application developers are telling you they want the kind of functionality that Docker offers. That makes it your job to figure out how that happens, not if it happens.

As long as me and my ops team are the primary people with skin in the game (read: getting 3AM wakeup calls), me and my team will maintain a say in what tech stack we're responsible for the care and feeding of. The dark days of dev throwing code over the wall to ops is long gone - I am gobsmacked that you'd suggest that this is for some reason improper.

What dev often forgets, and the cause of much strife, is this justification. If you want a team of n people to learn this new thing and its new ways of operating, then you need to do an honest, dispassionate cost/value analysis. And show your work, dammit. When this never happens, it conveys an "Oh great, Dev is on one of their 'Ooh, shiny!' kicks again" mentality to the rest of the team, especially those that aren't politically connected enough in the org to know any different. I know it's because you're busy and have insane deadlines to meet. Meanwhile, the phones ring at 3am, and the message is lost.

The result is a strained and adversarial relationship, often to the customer's detriment.

[thesuitonym]

I work in IT, so not exactly the same, but I can relate so much to your pain. Teams are constantly bringing in some half-baked app that "Worked in the demo" and give us the registration key and a date to have it running by. Invariably it become the biggest headache for us, because it never works, fails at the worst times, etc. And when it doesn't work, who is the finger pointed at? The person who bought into it wholesale without any testing, or the team who's been working 60+ hours every week trying to string it up with paper clips and rubber bands?

[jsmthrowaway]

That's exactly the same. Don't sell yourself short. You're doing operations with a different audience, and that type of behavior is just as common in external-facing operations. Hell, it doesn't even have to be an off-the-shelf product; I've been through that song and dance with in-house software.

[curun1r]

> As long as me and my ops team are the primary people with skin in the game (read: getting 3AM wakeup calls), me and my team will maintain a say in what tech stack we're responsible for the care and feeding of. The days of dev throwing code over the wall to ops is long gone.

Here's where we're in violent agreement. I believe the best way to ensure that the pager doesn't go off at 2am is to give that duty to developers. Far and away the most likely reason for a production issue is new code that went live. No matter how good communication is within an organization, ops will never have the same context for issues like this that dev will. So dev being the tip of the spear with regard to incident response will result in faster resolution. The job of ops is to be an enabler of this responsibility. They should be building tooling and infrastructure that makes it easy for dev to do the right thing but doesn't force decisions on dev. Ops is an abstraction layer that gives developers higher-level primitives for infrastructure that allows them to solve the customer problems more easily.

But there's no substitute for the dual benefit that you get from a) oh...I wonder if this problem is related to that change I reviewed yesterday and b) hmm...I've got pager duty this weekend, so perhaps I shouldn't just LGTM this PR. And for that reason, developers and developers alone are responsible for a working application. Ops is only there for support.

[jsmthrowaway]

> Where I get frustrated with ops is when they forget that they're in the service business. They're there to serve application developers in the same way that the application developers are there to serve customers. An application developer pushing back at a customer ask with the line, "without really stopping to ask what progress is from the perspective of developers" would be rightly lampooned. And yet somehow there's people that think the ops mindset is reasonable. In my view, "no" is not a valid viewpoint. Application developers are telling you they want the kind of functionality that Docker offers. That makes it your job to figure out how that happens, not if it happens.

Nope, nope, nope, nope. I quoted this entire paragraph at length because it's all just completely wrong.

Read Site Reliability Engineering that Google just published for an alternative perspective. SRE at Google is specifically and vocally organized to have authority to push back on engineering. Without exception. "No" is absolutely a valid viewpoint, particularly when you're asking a team to go on call for your code. It's not you sleeping in the guest bedroom and fucking up a marriage, now is it? No, it's us. Let's reframe this: you are responsible for software. I am responsible for production, which happens to run your software along with, likely, a bunch of other things that you don't know about. You said in another comment that operations exists solely to abstract infrastructure for developer consumption, which is such a heavy marginalization of what we do that I'm amazed you even identify yourself as an operator of any kind.

I'm not saying this to be an impediment to you. Both our paychecks flow from the same mission. I just need the authority to push back when your decisions threaten production (especially when you don't understand why). It's contrived, but on one end of the spectrum, if you said let's rewrite the stack in ColdFusion, it's not my job to figure out how to enable you. It's my job to figure out how to keep the lights on, and defending that properly means investing absolutely zero in your ColdFusion rewrite. Escalate to my management if you choose; if it comes down to the ColdFusion rewrite or my continued leadership of operations, I'll go somewhere else. So the more you ask operators to be "yes people" for engineers, the more you end up with operations outsourced to Tata and Infosys -- who will say yes to everything you want. Have fun with that.

Seriously, operations will be the equivalent of COBOL consulting by 2030. Just wait. I'll hit my mid-40s, pull a Mikey Dickerson, and build a consultancy to swoop in and clean up the mess that you're creating by marginalizing operations as strongly as you are and laugh all the way to the bank.

Particularly in SRE, I'm a software engineer who happens to enjoy the plumbing that keeps your code running. I do not exist to serve at the pleasure of application developers, nor do I accept any imagery of my being inferior because I don't spend my days writing red/black trees to shave a few milliseconds off the news feed. I can, and have, do/done SRE without a single application developer customer, and I like to think of SRE as a little bit better than application engineering: you push to a repo and wait. I push Enter and 10,000 things happen instantly. Who's having more fun?

Tossing lame code over the fence is how to sow bad seeds with operations. That you've then had negative experiences with operations, enough to generalize us as a species (are you hitting sysadmins? ops? sysops? devops? hwops? netops? SRE?), is only a reinforcement for that to me. I would say between a combination of educated guessing based on your opinion, prior experience, and a few other inputs that it's pretty likely that your gripes with operations stem from flawed expectations on your part.

[user5994461]

Rules #1 of operations: Don't be on call for other people shit.

You won't survive years in the field if you don't respect the basics.