[aj7]

“Let's say you notice that your company has a problem that I've heard people at most companies complain about: people get promoted for heroism and putting out fires, not for preventing fires.”

My first day at work at big-laser-company. Manufacturing engineer for a laser (then) so complex, it required a PhD to solve problems to get units out the door. The product was a ring laser. What that means is that the laser beam travels around in a race track pattern inside the laser before getting out, not a back-and-forth bouncing between two mirrors. Now this laser could be tuned to any wavelength by suitable setups and machinations, and once there, would “scan” a small amount about this wavelength, enabling scientists to study tiny spectral features in atoms and molecules with great precision. I knew all this shit. I was a Berkeley-trained physicist that built precision lasers out of scrap metal for my thesis. First day of work. I walk into the final test lab. The big laser was happily scanning away. The bright yellow needle-like output beam was permitted to hit the lab wall. As the laser scanned, the beam was MOVING on the wall. Whereupon, first day of work, I exclaimed the most obscene four words in manufacturing, for all to hear, “You can’t ship that!” (“Beam pointing instability” is detrimental to almost any laser application. It turns out that during scanning, an optical element was rotating, on a shaft, inside this laser. This mechanical motion caused beam motion.”) Well, I got an immediate reputation as a negative guy. (You can tell it’s deserved.) The solution was to retrofit 28 lasers in the field, mostly in Europe, with a component that cancelled the movement, on an expensive junket by a service guy. Who was hailed as a ”hero.”

[jacquesm]

One of the key questions in my due diligence practice is whether people are allowed to 'be negative' and to literally stop the line to avoid shipping a defective product.

This one question tends to separate out a very large fraction of companies that take unacceptable risks and allows the ones that don't to be justifiably proud of their attitude towards risk. These are not trivial things either, medical devices and software used in medical diagnosis, machine control and so on where an error can quite literally cost someone their life or a good chunk of their healthy life-span. Companies where people can not or won't speak up tend to have a lot of stuff that's wrong wiped under the carpet.

Kudos to you for speaking up, and irrespective of who got to be called a hero (that part isn't all that relevant to me) also kudos to your employer for acting on your input.

[Zagill]

Jidoka[1] is a key feature of Toyota's manufacturing process that emphasizes detecting defects before they make it out the door and empowering workers to stop the line and get to the root of the problem. It's weird that this isn't a no-brainer for most orgs but I guess there's enough profit incentive in shipping faster at the cost of quality.

[1] https://en.wikipedia.org/wiki/Autonomation

[bartread]

Yes, it's worked really well for them: https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehic....

I worked for a company that swallowed the (so-called) Toyota schtick hook, line, and sinker. About 14 years ago I tolerated some Toyota UK fossil coming in and berating me, in front of my entire team, for being a crap project manager, in spite of I was the most reliable and accurate product manager said (very successful and healthily growing) company had at the time. Seriously, still, fuck that guy with a nail-festooned cricket bat. I fucking shipped everything within the constraints I'd descrived at the beginning of the project, and it did great in the market. Anyone who doesn't like it is welcome to kiss my ass. But whatever.

Toyota or, more accurately, consultants who like to hawk the Toyota Production System (TPS), talk a good game, but the reality isn't always aligned with the ideals. Jidoka is evidently not a reality at Toyota, and they aren't much more enlightened than other orgs when it comes to pointing out problems, despite their A3 reports and multicoloured boards.

The Reckoning, by David Halberstam, makes it clear that "Toyota-like" practices aren't unique to Toyota amongst Japanese auto manufacturers. It also makes clear that these practices primarily exist to keep workers engaged and morale high (because, for those of you who've never worked on a production line [I have], in case there's any doubt in your minds, yes, it's boring as fuck).

The reason Toyota was much more successful than other Japanese auto makers in the second half of the 20th century is bugger all to do with their production process, and is instead the result of them being more aggressive and decisive in the wake of WWII: they simply opened a bigger factory sooner than their competitors and were therefore able to meet demand better. This gave them a trading advantage that lasted decades. The TPS didn't hinder their advantage, but it's absolutely disingenuous to claim it as the root cause.

Do NOT drink this koolaid about the TPS. I'm not saying there's nothing of value in it (I like genchi genbutsu, for example), but take it all with a pinch of salt. The value depends on who you are, who your team is, and how as a group you best operate. Fork-lifting business practices thoughtlessly from one organisation to another often doesn't work that well and TPS is no exception. It's no better than Agile cargo-culting but, because TPS is less mainstream, perhaps hasn't come under the same critical scrutiny.

Plus TPS's penchant for fault finding and negative culture overall just pisses people off and drags them down when they are (or should be) engaged in more creative problem solving. So something didn't work out: get over it, move on, and find another solution. Don't spend ages navel gazing about it. WTF? Seriously, if you think nitpicking everything and everybody makes you a good manager, you're an idiot and you should find another vocation. Fuck the fuck off. You're a tedious oxygen thief who's boring everyone.

Maybe it makes sense when you build the same thing over and over and over again, but we don't do that and we never did so it was always ridiculous to expect this to work well (and I say this as someone who, good faith, gave it a go, but the problem is that perhaps all the people pushing it at the time weren't acting in good faith).

[jacquesm]

As soon as something becomes a religion it loses most of its value.

To me the 'Toyota way' was more of an illustration than an exact guideline to follow and I've found this to be true for most of these things that tend to become a religion. Scrum, TDD etc all have this potential to become fodder for consultants that essentially sell a dream that they can not deliver on. But that doesn't mean there isn't a kernel of truth in there.

[hef19898]

Form over substance. Toyota, and other Japanese companies, are living the substance of Lean and TPS. Most other companies implement the form and hope they magically get where Toyota is without any additional effort. Same goes for Agile and any other management "philosophy".

[jacquesm]

Yes, it's the essence of cargo culting. The tech world is also full of this stuff. The number of small companies that I've seen that implement the Spotify development team structure is pretty tragic.

People are always looking for silver bullets and the industry is rife with examples of this kind of thing.

[bartread]

I'd like to apologise for the overly aggressive tone of this comment: I was drunk when I posted it. I have long since learned that drunk posting is a bad idea and yet, sometimes, I still am unable to resist that temptation. Anyway, this post isn't kind, and whilst I do have some issues with the TPS and think Toyota's history is often misrepresented, nobody needs to read a frothy mouthed rant about it.

[Zagill]

First off, name a single auto company that hasn't had a recall in its history. I think it's a bit unfair to point at a recall and imply that the systems they employ are bad because of it.

Secondly, more to your point, I'm certainly not trying to defend the whole system or even imply that it's effective at accomplishing its stated goals. I'm merely saying that the concept of encouraging employees working with/creating/designing a product to point out flaws and making a point of digging into where defects are introduced is a good idea. I definitely can't speak to how well that philosophy is applied at Toyota but I think that's moot regardless.

[analog31]

I don't dispute this, but at the same time, the cars from Toyota and Honda during the 1980s were vastly higher quality than their American and European counterparts. This could have contributed to the desire to emulate TPS.

[Cthulhu_]

Could it be an example of... well not survivorship bias, but, TPS gave them a competitive advantage in the 80's, and since then, other car companies have adopted it or something similar and upped their game, normalizing it to the point where people don't care about it anymore?

That is, once something is normalized you don't notice it anymore. Like how people that saw the 'rona epidemic was under control (ish) thought the measures were no longer needed.

[analog31]

I do think that people have much higher expectations today. Getting an engine rebuilt is mostly no longer a thing. The US mostly caught up, then Europe.

[varjag]

Same for their JIT supply management. The attitude I have a hard time mitigating with manufacturing contractors years after the global supply chain have deteriorated.

[jacquesm]

Those things work well when you're the only one doing them to gain an advantage over your competitors. But as soon as everybody starts doing it that means that the whole chain will adapt and suddenly all that stock that allowed you to do JIT and offload the costs of keeping that stock onto your suppliers evaporates which takes all of the slack out of the system. Now everybody has to perform and that will work right up to the first crisis and then the whole house of cards comes tumbling down.

It is always important to know what the underlying assumptions of your strategic advantages are. Going 'countercurrent' can work, but then if the tide turns you need to be aware that your previous advantage is now a risk.

[aj7]

Toyotas DO break less. But their designs are stodgy, very slowly updated, and generally draw premium prices. So customers keep flashier makes in business.

[MichaelZuo]

There definitely is a lot of pretend, or alleged, authority floating around in a large organization. But when it comes to brass tacks, the number of folks who really can do something and have it accepted is quite small.

[jacquesm]

In my experience this very much depends on two thing: the industry and upper management.

Some industries have a really lax attitude and in quite a few cases upper management basically makes it impossible for people to speak up.

[xyzzy123]

The flip side of "quality first" / stop the line is getting killed in market by a worse but faster or cheaper solution. You are locked in game theory with your regulators (if you have any) and your competitors.

It's disappointing (and career limiting) to do the "right" engineering and lose because you didn't correctly gauge the risk tolerance of the market.

I don't think there's any one answer for this.

My observation is that people will pay a premium for demonstrable physical safety features but privacy & security in software do not win markets.

[jacquesm]

This is mostly true but for things like medical devices, aerospace, industrial control and so on any corner cutting should be allowed.

I'm pretty sure that Philips right now has some thoughts on this as does Medtronic. Those two should have never happened and personally I'm all for liability of executives in such cases.

[jacquesm]

Should not be allowed, apologies. I always spot these errors only after the edit window has closed.

[aj7]

Heh, heh. They ignored my input and kept shipping. Then the customers began calling…

Meanwhile our major competitor cleverly placed their rotating element in such a position that the beam retraced itself through the rotating element, thereby substantially cancelling this effect.

[jacquesm]

Oh I totally misread that! From your story it seemed to me that they fixed it. Ok, that is less nice then.

I've played around with some - at the time - fairly high powered lasers and have extreme respect for them, the number of near accidents with those things was large enough that I learned to triple check everything and check for stray reflections at reduced power and the cleanliness of all optics before going all in. That saved me more times than I care to remember and is a nice reminder of how finicky a powerful beam of light can be. It doesn't take a whole lot to get a sizeable fraction of your beam ending up in places where you really don't want it to be. But they're lots of fun, even if they are dangerous :)

[smrtinsert]

I've also learned this as "bring the solution", regardless of whether or not you find the problem.

[actually_a_dog]

Do you actually use the phrase "be negative" when you ask the question? I could see that distorting the responses by people who don't consider pulling the cord to stop the line "being negative."

[quickthrower2]

Or … can the first officer order a goaround? Reading too much Admiral Cloudberg!

[geenew]

I once encountered a situation with a very expensive field laser. At one point, measurements started showing an increasing amount of offset.

Over a period of days, the error became increasingly, comically bad, until finally the system refused to boot.

A technician was called, and after hearing about the behaviour, the first request was that a photo of the laser light exit port be taken.

It was obvious why it wouldn’t boot: a mirror in the light path had fallen off.

The worst part was, the mirror had been held on by glue, and had been slowly slipping out of place. The hot climate was probably a factor.

They really should have had someone to say ‘you can’t ship that’ when the topic of glue to hold mirrors came up.

[pm_me_your_quan]

This exact problem happened with an optic in my lab in graduate school. For two years the senior grad student and postdoc blamed each other over the entire apparatus becoming misaligned every couple of days. (It was a really toxic environment.) Eventually, they both left, I was the only one there, and it still became misaligned. In one day I tracked it down to a prism from Thorlabs whose glue had gone bad positioned at the very beginning of the laser line- it was sliding in its mount.

I wish I had pushed more strongly about it. We spent probably a full person-day of work every week on that.

[whats_a_quasar]

Oh man, this one hurts to read a little bit. It's crazy how people cooperating poorly can eat up that much working effort.

[m463]

Reminds me of that giant pager outage ~ 20 years back. I remember one of the stories mentioned a woman who was going to leave her husband because he wasn't answering her page.

[fennecfoxy]

Well that's just abusive, really (to threaten to leave someone for not being in minute by minute contact with them/clear signs of an abuser power tripping).

[aj7]

Later in my career, I worked for a company whose principal technical strength was that they knew how to glue optics together in such a way that they NEVER moved, either thermally, or from shock. Detachment? The optic would break in an area besides the glue joint first. And the solution had little to do with the nature of which glue, which however was also optimized. These assemblies were flown in space, landed on the moon, and were in all U.S attack helicopters.

[c_o_n_v_e_x]

>They really should have had someone to say ‘you can’t ship that’ when the topic of glue to hold mirrors came up.

I work in product at a hardware company and have a lot of domain experience which came from spending years in the (literal) field. There's been many times where I write a product spec and the engineers are incredulous. "Really? It gets THAT hot?" or "Do we really need to provide a bonding/grounding lug on the case?"

It's not uncommon to find engineering teams with deep domain experience in one area, but completely lacking in others. Ignoring domain experience, there should have been rigorous product testing during design that would have weeded out the glue issue.

[KMag]

Interesting. I had an internship at a company that did inertial navigation, mostly for defence applications. I only knew of ring lasers for use in gyroscopes. (Send a laser around a loop wave guide/fiberoptic, and any translational acceleration cancels out going out and back, but any acceleration in rotational velocity in the plane of the ring/rotation vector perpendicular to the ring shows up as a Dopler shift. Tune the laser to have a standing wave, and rotational acceleration shifts the nodes of the standing wave around the ring.)

I had a colleague who got called up when a Trident missile MIRV bus fell off a forklift and he had to do simulations to tell the Navy if it was still good or needed to be brought back in for rework/recalibration. My understanding is that either the MRIV bus itself or its container has integral devices that record peak 3-axis acceleration for just such a scenario. I imagine they're as simple as a few precise weights on a few wires with precise failure strains, so you can bracket the peak acceleration by which wires broke and which survived.

On the one hand, it's great to have more accurate nukes, which allow lower yields, smaller stockpiles, and presumably smaller craters if everything goes sideways. On the other hand, "surgical" nukes result in it more likely that one side will use them and gamble that the other side won't massively retaliate.

[XorNot]

You could look at it a different way: a more accurate nuke means a nuke that's targeted at military facilities and not sized 10x larger and aimed at "everything around that city over there".

If it was ever used, that work saves lives.

[KMag]

More importantly, I think more accurate nukes along with good satellite multispectral and signals intelligence means that top generals carrying out orders for nuclear first strikes can be more certain that they're signing their own death warrants. Hopefully this results in any leader ordering a nuclear first strike getting deposed by military coup rather than starting a nuclear war.

[XorNot]

People are willing to die for causes all the time. The idea that a bunch of people would not take an action because it might kill them, particularly in the military, is pretty naive.

The history of nuclear brinksmanship is built on almost the exact opposite problem: people who are completely willing to sacrifice themselves for the cause and their government and who believe fully that the cost would be worth it and the decision would be correct.

At the expense of one's own life is one of the easiest sacrifices to make, and people who believe it are dangerous because they tend to volunteer a bunch of others to do so alongside them.

Nuclear command and control isn't about keeping any one person alive, it's very much about keeping the system functional so the deterrent is preserved. There's no way, within it, to actually ensure any level of personal survival - but the various advocates for first strikes at different points in history have never been concerned with that. They want their legacy, they want the problem solved "forever".

[tablespoon]

> More importantly, I think more accurate nukes along with good satellite multispectral and signals intelligence means that top generals carrying out orders for nuclear first strikes can be more certain that they're signing their own death warrants.

How would you do that? In the event of a nuclear war, my understanding is they'll mostly be flying around on special command and control planes. I don't think nuclear intercontinental SAMs are a thing. I'm not even sure if they could even be possible (wouldn't they need active guidance, which would be very hard on reentry).

[KMag]

Ahh, yes, the flaw in my optimism is that those doomsday planes do in fact have direct radio links to send the PAL codes and authenticated launch orders directly to the silos, submarines, and standby bombers.

The tier of generals just not senior enough to have a seat on the doomsday planes isn't in the emergency line of command to the nuclear weapons. So, regardless of how powerful a small coalition of those generals is, they cannot reliably prevent a nuclear launch. (They'd need a pre-existing conspiracy to quickly and efficiently turn their own air defence batteries against their own doomsday planes... at which point it seems very likely they'd just launch a coup long before a nuclear strike was ordered.)

So, I guess our last hope is that a small conspiracy of generals just under the doomsday plane tier would stage a coup once the nuclear sabre rattling reached a sufficient magnitude, before the nuclear first strike order is given.

[tablespoon]

> So, I guess our last hope is that a small conspiracy of generals just under the doomsday plane tier would stage a coup once the nuclear sabre rattling reached a sufficient magnitude, before the nuclear first strike order is given.

Even if that happened, it's just buying a little time. Some set of leaders/generals in the future will push the button (or build automated systems that do it for them).

Disarmament ain't gonna happen, and anything with a small chance of happening will happen, given a long enough period of time.

[throwbadubadu]

Please just don't do that. (Or missed irony?)

[thomastjeffery]

They proved you wrong, at great cost.

A more correct and polite version of your advice: "It will cost us a lot more to ship this as-is, and fix it later, than it will to delay shipment and fix it now. Is it too late to do that? Did we over-commit to shipping now?"

It wasn't your responsibility to come up with that version. It was your manager's responsibility. It was also their responsibility to find the necessary decision-makers and involve them directly. I would argue that this sort of work is the only real way that "management" can provide value in the first place.

Somehow, socially, it's incredibly common for people to value the inverse of that job. People assume it is "good work" for a manager to successfully ignore unpopular concerns, and push through to the end, no matter how inefficient that makes the journey.

That works out in the case that the shipping date was over committed, such that a delay would cost more than fixing it later. Even so, that entire situation would be avoided by refusing to over-commit shipping dates in the first place. That's the same responsibility applied earlier in time, so a manager that behaves the way I have described could factor out the entire problem at its source.

This is what the average person should learn about management. Even if it's not their job personally, there is a lot of leverage behind the decision a worker makes about what management behaviors to socially favor, and what behaviors to socially reject. That leverage is multiplied at every level up the hierarchy, making that the opinion of someone in a management role is very significant, and the opinion of someone in an executive role is crucial.

It's really difficult to be explicit about opinions. You can't really put them in your resume, but at the same time, an opinion on management style may be an executive's primary value contribution!

[LBJsPNS]

I have seen this far too often in engineering projects. The push is to get the product out the door and turn it over from R&D to production engineering. Never mind the final quality, that will be fixed in the field by another department with a different budget. We got a product shipped; our department's reputation and budget is intact.

[fennecfoxy]

I'm not in as prestigious a line of work as you are but I've found the exact same thing happens in my industry (web development, mostly).

Everyone wants to think of the cool ideas to make things work, but few people want to think of all the ways those ideas can break, fail, fail to be future-proof, be expensive, etc, etc whereas I relish in it; what's more satisfying than helping make a proposed or existing solution even better?

But the same applies to stuff outside of work, too. I find I'm quite negative about stuff in the exact same way and whilst it's fun to think "how could we fix this, how could we make it better", all people see it as is negativity and social pressure has made me start to rethink this approach in life. It's better to keep your mouth shut and let the fire start than to open your mouth and be negative, as per your analogy.

Hell it even applies to traditionalism; "we should put out that fire" "but that fire's always been there, that's the way it's always been" "but it's a fire!!!" "yeah, well it was here before you and we like it. That fire walked uphill both ways through the snow to get to school".

[hef19898]

Hah, that is so me! Well, not for lasers, but the dynamics are the same. You point out issues and risks, are ignored and labelled negative. Then when those things cannot be ignored, people come crying to you for help. And when they hear the solutions, aka stop doing what you are doing wrong, they again label you as negative, up to the point of blaming for everything that went wrong.

You are not getting points for preventing fires, you get them for putting them out. Unfortunately, some folks seems to conclude that lighting fire up, just to put them out later, is a good and easy way to earn that "hero" reputation.