[nonce42]

>The system worked this time.

I view the phrase "the system worked" as a red flag indicating that things went wrong in a way they shouldn't, not a positive thing. A successful outcome doesn't necessarily indicate things are fine.

A canonical example is the Space Shuttle boosters. The O-rings were supposed to block all the hot gases, but there was some gas leakage and erosion of the O-rings. Nothing bad happened and officials said this showed there was a sufficient safety factor. In other words, the system worked. They kept launching shuttles until an O-ring eroded all the way through and Challenger blew up. They should have recognized that erosion shouldn't have happened at all; successful launches despite erosion were an indication that things were dangerously wrong, not an indication that the system was working.

(I hope I haven't gone too far on a metaphorical tangent here.)

Reference: Feynman's report on Challenger: https://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roge...

[gen220]

As it turns out, there were in fact engineers who recognized that the O rings were eroding and that this was dangerous. They even tried to communicate this fact to the people with launch cancelling authority. But they weren’t able to communicate it effectively enough to prevent the launch.

Apparently, all of the engineers knew intuitively that the problem was real and critical, but weren’t able to communicate it to their superiors, who lacked the same intuitive grounding. They failed to visualize the data convincingly, and the launch went as planned.

It was an example of the system working, until people ignored the warnings about the O-rings. That decision to move on, rather than to investigate critically, was the moment when the system shifted from working to failing.

Edward Tufte wrote a really convincing treatment of this in “The Visual Design of Quantitative Information, and Envisioning Information“, to motivate the importance of proper visualizations.

You can read a blog with the visualizations here [1] but I really recommend the book!

[1] https://www.asktog.com/books/challengerExerpt.html

[dcow]

I think in this case, however, the system also worked as designed. O-rings aren’t designed to erode. The government, on the other hand, has designed power checks to prevent exactly what happened: a well meaning but ultimately idiotic decision from adversely impacting peoples’ lives in ways that weren’t initially accounted for. In no way am I giving credit to anyone here, just pointing out that the analogy doesn't quite hold.

[gen220]

If I remember correctly, the problem was that O-rings were designed to not erode at particular humidities and temperatures, which were subtly but critically different from the launch conditions.

This was actually observed by engineers at the time, who tried (unsuccessfully) to raise the observation to their superiors. The launch went on anyway, and the rest is history.

This has more details, along with my cousin comment: https://people.rit.edu/wlrgsh/FINRobison.pdf

[toby]

"well-meaning"? Are we still talking about the international student deportation order?

[andromeduck]

It's called defense in depth. Not everything has to be all over nothing.