[jiggawatts]

Please note the important caveat in the article:

If the following are true: Your team is following software engineering practices that have been shown time and time again to be indicative of highly performing technology teams (think CI/CD and DevOps).

This is all too often the root cause of developers being "too busy".

I've had some variation of the following conversation nearly verbatim with six different teams in the last twelve months:

Me: "I can help you guys to set up Azure DevOps pipelines to automate your builds and deployments, you just need to make sure your code builds on an isolated machine first."

Them: "That sounds nice, but we're too busy to work on that."

Me: "May I ask what is keeping you guys so busy?"

Them: "We have three (manual) deployments this week."

Me: "ಠ_ಠ"

[nstart]

While this example feels pretty obvious, I think in practice, these conversations conflict with the third caveat mentioned:

"Your team is always working on the next most important thing (as prioritised by the business)"

When the business functionality and marketing needs compete with technical advances, there's a decent amount of negotiation that needs to be done and that negotiation varies wildly from company to company.

Early on in my career I felt angry at these kinds of negotiations. As I've become more mild and open to business needs across the board I realise that most people will agree with the goodness of the idea but it becomes the duty of the proposer to sell it in terms that the business unit can agree to.

Overall, this is a great thought provoking article. Concise. Yet nuanced.

[KronisLV]

> When the business functionality and marketing needs compete with technical advances, there's a decent amount of negotiation that needs to be done and that negotiation varies wildly from company to company.

I think that this is the crux of the problem, but the solution lies elsewhere rather than just in negotiations.

After all, if you plead for some time to address technical debt and do platform improvements, you might just get told "no" to, given that in certain cultures people won't view these things as something that generates value directly and convincing them otherwise will be an uphill battle.

So why not just announce that X% of the following sprint or month will be spent on these things, to ensure successful continued operations? That's about the same as writing tests instead of having bad coverage due to an ever growing backlog and scope creep which eat up all your time.

After all, you don't ask business about using a version control system, do you? You just go ahead and do what's necessary to version your project. Tests, CI/CD, configuration automation etc. should be treated the same, since we're paid to be engineers and a part of that engineering is ensuring at least decent quality.

Note: this probably doesn't apply to larger efforts, like changing the entire architecture of your application etc.

[nstart]

I agree with the idea that solutions can lie in more than just negotiations. But I will also say that I’m at the point in my life[1] where I believe the vast majority of solutions do lie almost purely in some form of negotiation.

Take your own example:

> After all, if you plead for some time to address technical debt and do platform improvements, you might just get told "no" to, given that in certain cultures people won't view these things as something that generates value directly and convincing them otherwise will be an uphill battle.

This right here is a negotiation between business and engineering. The business unit cares about technical debt only if it affects successful delivery of business goals. As such it needs to be negotiated along those lines. In fact, I’d say you make a wonderful negotiation right after:

> So why not just announce that X% of the following sprint or month will be spent on these things, to ensure successful continued operations?

That is an excellent negotiation right there. The business cares about successful continued operations. Stopping everything to focus entirely on technical debt is not successful to the business. Allocating x% time to a mutually beneficial goal where x can be continuously tweaked is an acceptable point.

> After all, you don't ask business about using a version control system, do you? You just go ahead and do what's necessary to version your project.

The thing about this point is that it is:

A) easy to get setup.

B) well understood enough now to know the benefits to the organization

I’d argue that ci/cd is coming closer to that point now. Definitely is on the right trajectory and for the same reasons. Easy to setup and the benefits are well known and easily understood.

More nebulous topics like testing and tech debt will for the foreseeable future be topics requiring decent negotiation.

[1] I may change my mind on this in the future. Definitely been around on topics where I believe one thing, believe an alternate point of view, and later have a softer or accepting view of the original idea :).

[jiggawatts]

> well understood enough now to know the benefits to the organization

Maybe to you, but unless your business is 100% IT focused (like a software startup), I guarantee you that virtually nobody outside of the dev team knows what version control is, let alone the cost-benefit considerations.

A surgeon doesn't ask the patient whether to use antiseptic or latex gloves.

An engineer doesn't ask the client if the concrete should need steel reinforcing or not.

The lawyer doesn't ask the defendant if he should reference precedent.

The developer shouldn't ask the business for permission to automate their workflow.

[christophilus]

When I started at Microsoft, this exact scenario happened. I was told I couldn’t automate things because there was no time. I did it anyway, and a 3 person job suddenly became an extremely part time job for one person.

It blew my mind that anyone at Microsoft, of all places would have this mindset.

[TideAd]

When I was at Microsoft circa 2018, our team had a senior engineer whose primary job was doing manual validation of releases. That and keeping a single custom legacy build server running.

[wnolens]

I would love to share my experience of release management at MSFT but it would out me.

In the ballpark of 20ish engineers chasing build errors and test failures for a month to bump a version number (one line change).

[zo1]

Because it worked out for you, that time. As a manager and probably the person that would have told you no, or to justify more before we attempt it, I've seen this scenario go pear-shaped way more times than not. It's about risk, priorities, and specifically knowledge/experience (sometimes wrong) that says this is "not a good idea". Sometimes it's also just plain old complacency, though.

[robertlagrant]

It's amazing how many people you can hire (and how many bugs you can ship with) when you sell something that's free to distribute and has no true competition.

[bena]

I feel both sides of this one.

Yes, creating automated processes would make things easier, but the time it would take to automate the process would take too much time before you have to deliver.

So everyone knows what needs to be done, everyone recognizes that it would help things, but no one can put down what they're juggling to handle it.

Sometimes it takes bringing on a person just to take on that project as its own thing to relieve the pressure on the existing team.

[leghifla]

Regarding a dedicated person to handle the CI, here is my experience.

In the small company I am working now, I had setup a adhoc deployment script that was working fine, took less than 5min (with no user interaction) on my dev PC. Since it is not a SaaS, our release cycle was a bit slower than wanted, 1 or 2 times per month, depending on circumstances.

A guy was hired and he wanted to speed this up. I explained clearly that the build/deploy script was not the cullprit of the "slow" release cycle. It takes time to decide was is ready for production, write a nice changelog for users (not just collecting git messages), testing on the custom hardware...

That is why it took me an hour to half a day when the boss said: we need a release today. The above guy was justifying its work by: "After I am done, you will just need to put a tag and the rest will be automatic". I could not convince my boss this was fantasy land.

Result 3 years later: we have a "nice" CI which rebuild the world several times per day. But we are doing maximum 2 official releases per year, with much more stress. We had a few "releases" which needed very urgent hot-fixes because of last-minute changes and not enough testing on hardware.

And the CI person is constantly tweaking bits of the CI (it has become part of his job), breaking thing here and there.

I long for the old days...

[scatters]

Then I don't understand what your issue is.

> We had a few "releases" which needed very urgent hot-fixes because of last-minute changes and not enough testing on hardware.

That sounds like the problem is that the CI is not testing on hardware, or that you're running ad-hoc manual tests on hardware that could be automated or at least formalized.

[scatters]

Your problem is not the CI. It's that the CI does not test the custom hardware.

You need a hardware testing engineer who can automate the tests that you're currently running manually.

[leghifla]

The initial script was fine, but not ideal. The CI is better in this regards.

The CI is not the problem, sure. But it is not the solution either. That was my point.

[euroderf]

The strategic use of interns. With obvious ROI that makes subsequent interns easier to acquire.

[kqr]

There's also the more insidious version of this where instead they go, "That sounds nice, but we really don't need it."

"May I ask why you don't need it?"

"Oh, we found that we spent so much time on deployments that we decided to do only one planned deployment per year now."

In this case, the people you're talking to have obviously only swept the dirt under the rug, but sometimes they honestly think they've solved the problem this way!

(And then they ask everyone to "work harder" because the competition is running circles around them. This usually means cramming more things into already big deployments and failing even harder than before.)

[imtringued]

I always thought this was strange. A lot of our deployments don't take time to actually do, they mostly take time to decide when the next deployment is and what should be included. If I could deploy as fast as possible I think I would end up spending less time.

[kqr]

I'm not sure what you mean. If you have the luxury of deployments not taking time, then why do you need to debate what to include? Just deploy as soon as anything is ready for deployment.

[jiggawatts]

"Normalisation of deviance" is a wonderful thing to Google.

I've seen people trying to justify a workflow what is literally 10,000x slower than the industry norm. Then patiently explaining to me that dealing with the fallout of that is necessary, and I was just being uncooperative.

A recent example is I showed a dev team how to identify and fix some crash bugs in minutes. Because they were "busy" and deployments require a mile high pile of paperwork, six months later those bugs are still occurring in production.

They're busy with BAU / support tickets, which is why they can't get around to doing a deployment of bugs they've already fixed.

Guess what the users in those tickets are complaining about.

I think they've fixed the same bug in like.. three separate un-merged, un-deployed branches.

This is what the opposite of CI/CD looks like.