[smallnix]

In theory a good escalation system. In practice there must be strong guarantees and trust that there are no repercussions for triggering. Otherwise management will tell you over and over to "pull the andon cord / escalate earle & often" but really no one does.

[kqr]

Not only must there be trust, but there must also be a resolve to make deeper fixes to problems surfaced through the andon system. It's a typical mistake to forget that part. If the response to a problem is patching over the immediate symptom, then the cord will keep being pulled for the same reasons so often no work will end up getting done.

Then the andon system is abandoned as "didn't work for our organisation".

Workers should pull the andon cord (and thus stop the entire production line!) when they need to go to the bathroom, for example. The solution is not to sternly tell the worker to hold it for longer, nor to have a replacement worker come over, but to review scheduling and include more appropriate bathroom breaks between shifts at the line. (Or, if the problem affects one worker disproportionately, figure out some alternative way for that worker to contribute.)

[jeffrallen]

Yeah, at $WORK, the incident doc has a section for corrective actions. And they mostly get done, but having them struggle for priority with regular work is a problem.

Also, it occurs to me that we leaders should do a better job of saying, "look at that excellent job of attacking that corrective item that team just did!" so that it's clear to people that this is not clean-up work, but in fact the most essential kind of tech debt that's actively being paid down, which identified itself as such by causing an outage.

[niam]

It's not just resolve, either. The org must recognize the problem in the first place, which isn't a given. Especially in smaller/govt orgs. Often, recognition of the problem is wrongly tied to how difficult the foremost suggestion is to implement.

And there are orgs for whom any suggestion can itself be so encumbered by uncontrolled red tape and social costs that relatively minor changes become a project.

[Cthulhu_]

It only works well in a "no blame" culture - just like software development should be. Sure, one person can really mess up, but there should be systems and processes in place that prevent that from happening or becoming a major issue. This includes limiting access to production, code reviews, CI, etc.

It feels like these best practices are often forgotten or skipped by people who just want to feel productive (be it through writing their own code or using AI). Which is fine to a point - for personal projects.

[jgilias]

Yes. There was an article somewhere about how in a Japanese factory it gets pulled thousands of times a year, but only 2 in an American one.

EDIT: Found it: https://davidoks.blog/p/why-japanese-companies-do-so-many

Search for “Ford plant”, second occurrence for that particular bit. The article made rounds on HN a couple months ago.

[collabs]

Maybe it works for small stuff like running out of bolts in a production line but not something high level like the owner is an idiot and made a massive mistake. I sometimes think about the high profile Amazon Fire Phone and why nobody involved in building the Amazon fire phone say at any point that this thing was destined to fail. I'm sure Amazon dot com had highly intelligent workers and managers who saw what was coming but never spoke up.

I have never ever heard from any Amazon employee I've met in person tell me of any instance where they told their manager or supervisor something and had the superior "disagree and commit". It always goes in one direction, down unlike what they say in their HR material.

I don't think this is a solved problem at all, short of making it very inexpensive to pull that proverbial cord as a worker AND making it very expensive to ignore such cord pulling as management. I don't know if it is possible to have that with our management system today. These two properties — cheap to pull and expensive to ignore — are intertwined. It means management giving up a lot, perhaps almost all, of its power to the workers. If you follow through with this, you also need to share more information because if the workers are actually empowered to decide, they should have the information necessary to make such decisions.

It requires someone with power to consistently and deliberately eschew this power which isn't sustainable because at some point in the management chain you will come across someone who will not.

Even at Toyota, I don't think you can pull the Andon cord on hydrogen fuel cell to switch to electric vehicles because at this point you are not just up against management, you are up against national energy policy.