[gtmitchell]

A very thorough preliminary report. I've worked for a long time in quality systems, and this is a perfect example of a systemic failure. They've got work being handed off between Boeing employees and 3rd party contractors with insufficient controls in place to verify that very basic tasks are being performed.

I'd be curious to know how many non-conformances they typically see during assembly of a plane and whether management is actually allowing the quality department sufficient independence to investigate these issues and fully resolve them. I'm guessing that the production personnel are under tremendous time constraints and are constantly pressure the quality assurance people to sign off on whatever paperwork is holding up the line, no matter the safety implications.

Also, I think a lot of middle and upper level management needs to lose their jobs over this. I hope this mess ends up in textbooks and gets beaten into the head of every MBA student in the country.

[onetimeuse92304]

> I'd be curious to know how many non-conformances they typically see during assembly of a plane (...)

Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible.

For example, better quality control usually results in an increase of number of defects, at least temporarily. But that just because large portion of these defects were undetected before.

So... you are looking at a number that you have nothing to compare to that also depends on how closely the process is monitored and also depends a lot on the definition of what is non-conformance.

It is like trying to give an answer to "what is the length of Britain's coastline?" Everybody knows that you can get whatever answer you want depending on how long the ruler is.

[outworlder]

> Very likely that number is meaningless. I suspect this is the kind of environment that incentivises hiding non-conformances whenever possible.

That, by itself, should be the kind of thing that should shutdown a company permanently. Remember, this is the aviation industry, where they track the mine where the ore from the bolt was mined, and who tightened the bolt up and with which torque wrench.

[whythre]

Sounds like a lot of steps and paperwork… steps and paperwork that might easily and regularly get fudged.

[l33tman]

The idea is that the system as a whole should be resilient to a certain degree of this, as it will be caught in other places or at least reported at a (hopefully non-fatal) aviation incident that leads to a report and analysis, at which point you backtrack and issue recommendations that actually have clout (like, you can't fly with your plane anywhere in the world if you don't fix this).

Of course if the entire industry is corrupt it doesn't work, but to a certain degree I guess it is robust. This time it led to a potential disaster but it will become safer as a result..

[onetimeuse92304]

Of course. And the way to go is to set up incentives so that everybody wants to report issues rather than hide them.

In a normal, healthy situation a company like Boeing should not feel threatened if some problems are exposed from time to time. That is assuming that everybody understand that uncovering these problems is part of the process and is necessary to improve safety and is exactly why and how we have good safety in the first place.

It only becomes a problem when that safety record becomes blemished too much.

I am pretty sure it actually has worked successfully for many decades up until some point, evidenced by consistently improving safety record.

[javawizard]

This sort of thing always reminds me of this Dilbert comic: https://twitter.com/k8em0/status/1078824013843984384

[wmidwestranger]

> Also, I think a lot of middle and upper level management needs to lose their jobs over this.

Given that if a worker doing the work raised this problem or took initiative to resolve it, they'ld probably be punished; I completely agree.

Reminds me so much of how, once-upon-a-time, there seemed to be actual engineering management in a cooperatively adversarial relationship with business managers but not anymore. Now any sort of engineering in business seems to be completely business managed and business minded. I'm sure it's great for profits while it lasts but I haven't observed engineering becoming better and I suspect business is suffering by overextending itself too, I just don't have any solid observations.

(Well, maybe one, my brother does warehouse / logistics management and says, despite there being every reason in the world, he has never seen the accounting software and the inventory software successfully and productively linked. So, big opportunity there for a serious player but maybe not the profitable compared to the issue?)

[wolpoli]

Once upon a time, QAs and developers also had a bit of cooperatively adversarial relationship as well. It's no more now as teams are restructured by business, so they cooperate.

[shadowgovt]

Rumor has it the controls are there, but subvert-able.

Apparently, there are two ticketing systems (one for "history of plane," Boeing internal, and one for "day-to-day onsite work," visible by contractors and Boeing management). The work to fix the rivets was logged in the day-to-day, but management and the onsite staff managed to convince themselves that merely opening the plug to fix the vacuum-seal trim did not constitute "removing" the plug, and since there was only an entry in the history-of-plane log for removing, not opening, they didn't log it there (when the intent was "there's no entry for 'just opening' because there's no such thing as 'just opening', breaching the pressure vessel at all constitutes 'removal of plug'").

The final inspection that should have caught the error would have been triggered by the update in the history-of-plane ticketing queue.

(And as for 'how many non-conformances,' the same source claims that Spirit is one of the few subcontractors with on-site staff at the factory because their parent company delivers such consistently shoddy out-of-compliance product that they are continuously doing final warrenty-work onsite. So maybe "fire that vendor" should be on the docket too).

[ethbr1]

You might want to cite your source on this, which I'm guessing is the purported insider speaking about same?

From the portions of the report that hint at corroboration,

> Documents and photos show that to perform the replacement of the damaged rivets, access to the rivets required opening the left MED plug (see figure 15). To open the MED plug, the two vertical movement arrestor bolts and two upper guide track bolts had to be removed.

> Records show the rivets were replaced per engineering requirements on Non-Conformance (NC) Order 145-8987-RSHK-1296-002NC completed on September 19, 2023, by Spirit AeroSystems personnel. Photo documentation obtained from Boeing shows evidence of the left-hand MED plug closed with no retention hardware (bolts) in the three visible locations (the aft upper guide track is covered with insulation and cannot be seen in the photo)

[MBCook]

Yes what they said about two systems was from the insider account that was posted online.

[fransje26]

https://leehamnews.com/2024/01/15/unplanned-removal-installa...

and

https://leehamnews.com/2024/01/15/unplanned-removal-installa...

[lp4vn]

>I'd be curious to know [...] whether management is actually allowing the quality department sufficient independence to investigate these issues and fully resolve them

If management in the aerospace industry works like management in the software industry, then I guess they are pushing for results as agressively as possible without much concern about safety or anything else.

[cezart]

At a company I worked in, we had a joke about this: "Good thing we don't build nuclear reactors".

In some software projects the level of rush, and the fact that bugs sometimes would leak into production was kinda horrifying. It would've been way more so, if it would've been the kind of project that could kill people in case of failure. Like it happened in Chernobyl with nuclear reactors, or at Boeing with planes.

I can't really imagine what these engineers feel when they rush this kind of work knowing what's at stake.

[toast0]

Quality is always a trade-off. If you're deeply into economics, you wonder about the trade-offs of cost to find defects before shipping, difference in cost of addressing defects before and after shipping including costs of mitigation from consequences of defects, % of defects that will never be found after shipping (and are therefore a real cost savings), and in the long game costs of having a reputation for shipping product with defects that could have been reasonably detected.

In a lot of software organizations with rapidly changing and undocumented requirements, there's a good chance defects will go unnoticed until they're no longer relevant, so spending a lot to find them before they're shipped is a waste. Mitigation of many software defects is simple, but some aren't; hopefully you know which changes are expensive to fix if wrong, so you can more thoroughly vet those.

In Aerospace, addressing defects after shipping is very expensive, and mitigating the effects of defects is only approximate; you can't restore passengers from backup, economic damages don't really make families whole, but should be an incentive not to let reasonably detectable defects be shipped.

[wmidwestranger]

> Mitigation of many software defects is simple, but some aren't; hopefully you know which changes are expensive to fix if wrong, so you can more thoroughly vet those.

This assumes you're fortunate enough to have a defect at the outer edge of the system. Most times, these problems are created in the initial rush of pushing something out and then tax every effort that depends on them, forever, and ever.

[masfuerte]

Amen.

[lp4vn]

>In a lot of software organizations with rapidly changing and undocumented requirements, there's a good chance defects will go unnoticed until they're no longer relevant, so spending a lot to find them before they're shipped is a waste.

It's really a shame that a good percentage of these applications full of bugs and "rapidly changing and undocumented requirements" don't get scrapped and stay many decades afloat until they get replaced by another application also full of bugs and "rapidly changing and undocumented requirements".

I think that that's a very sad way of seeing things honestly.

In the past the USA put the man on the moon, today repeating the same feat looks almost impossible. I bet that a lot of managers at Boeing also think that building planes like a few decades ago looks almost impossible now.

[bbor]

I really find it disingenuous to imply that we can’t build moon rockets because we’re not good enough at engineering projects - I can think of a few engineering projects that took off last year, to say the least. And nasa doesn’t deserve the shade IMO. The kids of the people who built the Apollo program aren’t working at boeing or fighting for one of the few underpaid nasa positions - they’re building reusable rockets for the Twitter CEO, and, much more commonly, parasitic UX features for gig economy apps.

TL;DR: we’re fine at engineering, we’re terrible at resource allocation. Or at least that’s the more relevant cause. I post this knowing full well that this is HN and I might well be disagreeing with a senior nasa employee…

[TheOtherHobbes]

If we're fine at engineering, why are doors falling off planes?

I don't think you can say "We're fine at engineering but we're often terrible at management."

They're not separate things.

Engineering culture is about inventiveness, pride, craftsmanship, and getting the job done well. Bean counter culture is the opposite of all those. If that's the culture engineers work under not only do none of them happen, but they become less and less possible over time.

[maxerickson]

Quality is always a trade-off.

That's a pretty huge assumption.

For instance, compare firearms prior to replaceable parts to firearms after.

Better, cheaper, easier to make (because craft was replaced with process). Some up front cost, but absolutely not a trade off, it was a huge advancement.

Of course modern process control does more or less let you relax conformance rules to reduce cost, but it's farcical to call sacrificing reasonable conformance "quality".

Arguably, the idea that quality is obviously a trade off and you can make money by letting it slide is one of the sources of rot in our society.

[sokoloff]

> Some up front cost, but absolutely not a trade off, it was a huge advancement.

Any exchange of higher fixed costs for lower marginal costs (or other benefit) is a tradeoff.

This is a tradeoff that was/is massively beneficial, but it’s still a tradeoff.

[maxerickson]

It's not a quality trade off, which is at least implied to be what I am talking about in my comment.

Saying "Quality is always a trade-off" implies you can't actually make things better over time. I would accept something like "You can always trade off of the quality you are capable of producing to reduce costs", but the point is that capability can in fact change, and thus there are ways to improve quality that may not be a trade off (because they also improve the business/system along other dimensions). Even simple little things like aligning a process with the intended outcome can reduce costs while improving quality, you don't have to invent a revolutionary method of manufacturing.

[Turing_Machine]

As the old saying goes: "Fast, cheap, or good. You can pick a maximum of two."

[trompetenaccoun]

That's not how it traditionally worked in aerospace. Commercial airliners are such a safe mode of transportation because from start to finish there are (or used to be?) very high safety standards all around. I can't speak for what's going on with Boeing currently though. Things certainly seem to have deteriorated.

[mihaaly]

I have tendency to hope for unrealistic things too. Not a reliable trait of mine, no. But in my clear moments I am afraid that much sooner will come the world peace and union of all nations and religions than unimaginative but determined bean counters learn from millions of catastrophes of the past and future to come giving up pushing their core value and first rule of 'take more, give less' into the infinity and beyond. And giving up their personal wealth with it.

[zmgsabst]

I think the hope is people of good conscience stop being political cowards.

If you believe in quality engineering, but refuse to engage in the political and business dimensions of an enterprise to fight for that view, then you are just virtue signaling — since you’re refusing to engage with the tools needed to make it happen.

[mihaaly]

It is always refreshing to meet fellow naives like yourself, I like the company of the likely minded!

Engaging in today's political theatre and culture (that is the keen slave of forceful bean counters/collectors btw.) with the prediction of constructive advancement for the quality of life, that is some very hardcore stuff! Even for me.

(what I can do beyond raising voice is to show example rather than piss into the headwind hurricane. and hope others will follow, or I can follow. but definitelly not replacing my inner Gandhi with Rocky in a blink and getting into a fistfight with scores of agressive pro boxers simultaneously who are itching to run over anyone in their way. also I can avoid the products of those bean people, not feeding them with my purchases or assistance any way, encouraging others doing the same. vermins should starve, not bloom!)

[zmgsabst]

I meant in the practical senses:

- learn to make your arguments in terms that appeal to the sensibilities of bean counters, not engineers

- learn to manipulate documentation so they’re forced to record overriding good engineering in a way that’s discoverable

- learn to identify key political players and address their desires, rather than appealing to “doing the right thing”

Etc.

All things that I’ve done poorly at various stages of my career — and seen other engineers struggle with as well. Much like being a manager requires training and education, so does being an effective advocate for engineering to the wider organization.

Or to borrow your analogy, if every engineer is afraid of boxing, then can we be surprised their views go unheard?

[lazide]

First, being cowards this way would need to stop being the wise choice to make.

[pooper]

At a minimum, we must abolish the right to work at a federal level. Sounds so terrible, doesn't it? Pooper is against the right to work. Pooper wants to take away jobs. No, we can't have that because of politics. And thus, we are stuck.

[fransje26]

According to the "insider source", 392 non-conforming defects in the fuselage door installation in the last 365 calendar days.

> As a result, this check job that should find minimal defects has in the past 365 calendar days recorded 392 nonconforming findings on 737 mid fuselage door installations (so both actual doors for the high density configs, and plugs like the one that blew out). That is a hideously high and very alarming number, and if our quality system on 737 was healthy, it would have stopped the line and driven the issue back to supplier after the first few instances.

Source:

https://leehamnews.com/2024/01/15/unplanned-removal-installa...

[michaelt]

> I'd be curious to know how many non-conformances they typically see during assembly of a plane

Well, the report says "During the build process, one quality notification (QN NW0002407062) was noted indicating the seal flushness was out of tolerance by 0.01 inches.

So I'd say they've had about 2,407,062 quality issues :)

[Arrath]

Then its revealed "Well the numbering schema started at QN AXX... and we looped through the alphabet a few times.."

[hipsterstal1n]

I was speaking with some friends at a Christmas party who work for the Navy - they’ve taken deliveries of planes from Boeing with the same sort of issues that start in the factory. They even went as far as to say the whole lot of planes should’ve been rejected but weren’t. Multiple things not built to spec.

The planes they worked on did not share an assembly line with the 737 but another Boeing model…

[fransje26]

If I recall properly, sometime after the MAX crash saga, the Air Force simply refused to take delivery of their 767s because of unacceptable build issues, including foreign debris (aluminium rests and shavings) and forgotten tools flying around.

I didn't follow how that issue evolved.

[TurkishPoptart]

"....after the left mid exit door (MED) plug departed the airplane leading to a rapid decompression"

Lol, they said the door plug "departed" instead of "blew the f* off"