[aftbit]

>Doing “DevOps work” is unquestionably the worst part of building a modern application. It’s full of tiny papercuts, indignities we suffer in our toolchains, our feedback loops, and our software. It’s a city of brutalist buildings filled with sharp-edged couches pretending to be comfortable. Think of all the advances in how we interact with tools in other domains - then take a look at the way you build, deploy, and operate your software, at all the crazy gyrations you use to glue it all together - and ask yourself why you accept it.

I dunno, usually I find databases and migrations to be the hard part. At this point, I have enough examples of app deploys that I can have a new app up and running on a pair of VMs with a robust blue/green deploy and backups inside of an hour or two, with deploy by Github Actions responding to pushes to prod branch.

Even if you don't have my company's half-decade worth of example devops, you can do something easier, like a single instance on a Digital Ocean machine with deploy by "ssh -A server 'cd yourapp && git pull && sudo systemctl restart yourapp'". Sure, you'll have a few seconds of downtime, and you'll expose your SSH keys to anyone on that box for those few seconds, but if you know some Linux and nginx, you can get this working inside of an hour from scratch.

[hinkley]

Fundamentally, I think the “why” is still the fact that DevEx started with devs, and DevOps started as a collaboration with Ops people, who already were not speaking the same language of robustness that we do. Which is a little weird, and probably part of the friction between the groups.

They historically took on the reliability role, if nobody else did, but they were implementing reliability on top of a house of cards, which is a kind of hypocrisy that makes even mediocre devs bristle. Don’t lecture me on robust software, boyo. Your tools are made of string cheese and staples.

[pmoriarty]

"They historically took on the reliability role, of nobody else did, but they were implementing reliability on top of a house of cards, which is a kind of hypocrisy that makes even mediocre devs bristle. Don’t lecture me on robust software, boyo. Your tools are made of string cheese and staples."

I don't know why you'd blame ops for the crappyness of the tools they have at their disposal.

Yes, Ansible, Salt, Puppet, and Chef are spaghetti-code inducing congealed messes of design. So are large collections of complex shell scripts.

So what's the alternative? What spherical cow of a configuration management tool from Platonic dev heaven shall be foisted on us this time?

I'm sure it'll be super clean and elegant this time, unlike the last thousand shitty tools they made.

And don't get me started on devs that think they're qualified to do ops when all they know is their language of choice (if even that) and have never thought about the network, security, capacity, redundancy, failover, reliabililty, hardware, backups, the rest of the company or other users.

[jrott]

No no no Kubernetes or Serverless or ChatGPT is going to save us this time.

More seriously it always going to be complicated and annoying. It's really past time we started dealing with the fundamental complexity of everything we are trying to do with software.

[Spivak]

Because you're not talking about that thing that ops people are talking about. We build reliable systems. You're talking about reliable software. Ops people come from the perspective that all software is inherently unreliable including your app, especially your app and have to work within those constraints.

Terraform and Ansible look like gyroscopes compared to the build process of any modern software stack. We offered our dev teams a whole ass pizza party every time they had 10 green builds (on main) in a row. In three years we've paid it out once.

[hinkley]

The operational tools should be the most stable bits and instead they are janky as fuck and I’ve spent too much of my career smacking victim-blaming tennis balls back over the net. If you look at what Ansible replaces it’s a wonder production ever worked at all. If you have to baby your automation it’s not automation.

Ops people are not used to thinking in boundary conditions. Hell, devs forget half the time. That’s part of why people wanted to merge them in the first place. Get the right sorts of cynicism together in a room and make me something with a big green button an idiot can push while everyone is in a meeting.

[thezilch]

Oh boy, a whole pizza!

[hinkley]

Bad for both vegans and gluten intolerant people. Also pizza parties are sooo 2006.

Nothing demotivates people like a half assed reward. If you want something never to get done, offer a bad carrot. If you want only one thing to get done and nothing ever again, offer a good reward but then make the recipient fight you to get it. I thought pizza parties were the worst, and then I got an award that was announced but not delivered, and spent almost eight hours over the next month poking people to give me my goddamned prize. It became a second job and at the end I ended up resenting the entire experience.

Later at another job when we in theory had a generous reimbursement process that was actually a ton of red tape, I had flashbacks to that incident.

[Spivak]

No, a full on everyone take the afternoon off catered food drinks to watch movies and play video games in our offices's bar paid for by the ops team personally.

Also don't assume we don't know our coworkers, when they first got to 8 in a row the slack channel looked like the Twitch chat of a popular streamer.

We have the buy-in from management that if it happens every day then it happens every day.

We are planning to change up the food if it ever happens again, pizza party is just a good title

[zsoltkacsandi]

My experience is (as a former dev, current ops) that the problem isn’t that dev and ops people aren’t speaking the same language, nor the tooling or processes.

The problem comes from the management/business side. They hire devs and tell them that ship features as fast as you can. Also they hire ops guys and tell them that I want this whole thing super reliable, we can’t afford a minute downtime.

In my opinion this is why DevOps is mostly pointless. We are trying to fix with tooling, processes, new tech, and fancy roles the fact that business people don’t want to make compromises or choose between the pace of delivery and reliability.

[hinkley]

That’s definitely part of the dynamic. My biggest regret with automated testing is that software used to be a triumvirate of Quality, Dev, and Management, and when dev was fucking around they had two teams hitting them, and when Management was out of control, they had everyone mad at them. Get rid of QA and it’s Us vs Them and that worked briefly at the dawn of Agile but they got wise.

OP’s should replace QA at that table to rebalance the equation. But again, and as you illustrated, we have an adversarial relationship that takes a lot of across the aisle work to introduce sanity.

[convolvatron]

I agree. but having a QA group was hardly a magic bullet. most of the really talented contributors eschewed QA - even if they enjoyed it - because it meant being identified as a QA person for the rest of their career.

even if you could staff a competent group, they are often left with nothing to do while the devs 'work their magic'. and suddenly its 3 months past the original 'functionally complete' deadline, and QA is given 1 week to do what they .. oh, maybe 3 days really, to do what they need to do.

when it works its irreplaceable. when it doesn't its just a lot of noise and spend and flailing.

[cmrdporcupine]

The most productive QA/dev team experience I had back in the day was in a shop that was playing with eXtreme Programming and we did little 2-3 person strike mission per-issue groups with 2 to 3 devs and 1 QA person. Work on strictly defined small feature and continuously QA through the process, and then iteration end the QA person worked more intensely. The point being that the QA staff was treated like another development engineer even if they weren't writing code, so no "toss this over the fence to QA" and "oh god this is horrible, the devs are shit" mentality. Outcome of the feature was not just completed code, but (hopefully) test plans and sign-off from QA.

The QA people hated it though.

[hinkley]

I would definitely say that some of the finest and worst people I’ve worked with were in QA. Mercifully usually not at the same job. It can be a great place to push ideals, or to hide from them.

I know why it’s gone, I’m just not happy about it.

[bob1029]

> Even if you don't have my company's half-decade worth of example devops, you can do something easier,

If you are open to it, try configuring an azure function app to use GitHub in the deployment center. I heard actual gasps from certain team members when they saw it automatically push the GH action workflow file into master and kick off the job without any additional bullshit beyond the GH authentication ceremony and org/repo/branch selection.

[giovannibonetti]

> I dunno, usually I find databases and migrations to be the hard part.

For me, the following tools make that a joy: - Postgres as the database, which is very predictable and extremely reliable; - Migrations with Ruby on Rails, that have just the right balance between a convenient DSL and letting you write SQL when necessary; - The strong_migrations gem that catches in development unsafe commands to run in production, and explains how to make them safe

[efxhoy]

We run the exact same setup, in my two years at the company we’ve only had one migration related issue and that was due to different minor versions of postgres between CI and staging. Now if postgis could get those official ARM docker images pushed i’d have nothing left to complain about.