[kqr]

Not only must there be trust, but there must also be a resolve to make deeper fixes to problems surfaced through the andon system. It's a typical mistake to forget that part. If the response to a problem is patching over the immediate symptom, then the cord will keep being pulled for the same reasons so often no work will end up getting done.

Then the andon system is abandoned as "didn't work for our organisation".

Workers should pull the andon cord (and thus stop the entire production line!) when they need to go to the bathroom, for example. The solution is not to sternly tell the worker to hold it for longer, nor to have a replacement worker come over, but to review scheduling and include more appropriate bathroom breaks between shifts at the line. (Or, if the problem affects one worker disproportionately, figure out some alternative way for that worker to contribute.)

[jeffrallen]

Yeah, at $WORK, the incident doc has a section for corrective actions. And they mostly get done, but having them struggle for priority with regular work is a problem.

Also, it occurs to me that we leaders should do a better job of saying, "look at that excellent job of attacking that corrective item that team just did!" so that it's clear to people that this is not clean-up work, but in fact the most essential kind of tech debt that's actively being paid down, which identified itself as such by causing an outage.

[niam]

It's not just resolve, either. The org must recognize the problem in the first place, which isn't a given. Especially in smaller/govt orgs. Often, recognition of the problem is wrongly tied to how difficult the foremost suggestion is to implement.

And there are orgs for whom any suggestion can itself be so encumbered by uncontrolled red tape and social costs that relatively minor changes become a project.

[Cthulhu_]

It only works well in a "no blame" culture - just like software development should be. Sure, one person can really mess up, but there should be systems and processes in place that prevent that from happening or becoming a major issue. This includes limiting access to production, code reviews, CI, etc.

It feels like these best practices are often forgotten or skipped by people who just want to feel productive (be it through writing their own code or using AI). Which is fine to a point - for personal projects.