[shangxiao]

> It is funny people can't envision things breaking...

This is what enraged me about the Therac-25 design which ended up killing people. There's a whole report to read on the design, failure and follow up but to paraphrase the designers eliminated the physical locking mechanism and had this attitude that "oh it's electronic so it won't wear out and fail".

[pmontra]

Therac-25 https://en.wikipedia.org/wiki/Therac-25

Software errors killing people with excessive radiation doses.

[Thiez]

It's a bit sad to see it coming up to regularly. On the one hand it's a very classic example of programming bugs leading to injury and death. On the other hand it's so long ago that bringing it up so often makes it seem like it's the only good example of dangerous software. Software must be pretty amazing and safe that we have to keep bringing up an example from the late 80s.

[ilyt]

Uh, no. We have plenty of examples, they just don't kill people that often, because some industries learned their lessons.

If you want recent one, 737 MAX, killed way more than Therac-25. Although it was as much fault of software as suits around it wanting to save a penny on training, because if pilots knew about how it exactly worked they could've circumvented it

[error503]

The software on the MAX worked as designed/specified; unlike Therac-25 there was no bug in the critical path, it was a series of design and oversight failures pushed by business and cost cutting interests, and the actual accidents were triggered (though I wouldn't consider it causal) by hardware failure in one of the AoA sensors. There was a bug regarding displaying an AoA disagree warning to the pilots, which despite being known wasn't fixed by Boeing, but this wouldn't have actually changed anything about the plane's behaviour.

To the credit of systems engineers, I can't think of a recent high profile fatal accident that could be reasonably blamed primarily on software, but that's not so much because software is infallible, but because systems are designed to fail safe.

[ionwake]

Correct me if I’m wrong but wasn’t there an overflow in the software if the aircraft climbed too steeply ?

I’m sure I read that

[error503]

I don't think so? Maybe a bug of this form was found, but I'm sure nothing like that was involved in the crashes. The flight control software performed as it was designed to, it wasn't software that sent the trim wheels spinning, but a bad AOA sensor and a lack of proper safety analysis, training and procedures.

There was a bug that caused the AOA DISAGREE alert on the EICAS not to be displayed, because at some point someone misunderstood the requirement that the AOA indicator should be hidden if they didn't pay for the upgrade, but this was just an indication and wouldn't have affected control at all (though likely would have hinted the pilots to a more appropriate cause of action).

One could also consider the lack of cross-checking between the two flight computers and associated AOA sensors to be a bug, but that was how the system was intentionally designed, because AOA wasn't considered a flight-critical measurement in the system's safety assessment, so they didn't consider this required. A holistic safety analysis was never really done inclusive of MCAS though, and this requirement probably just followed on from 737NG and wasn't really considered (at least thoughtfully...) in MCAS' design.

[ilyt]

The main bug was:

* each flight computer used its own angle of attack sensor with no way to detect failure. Craft had 2 sensors, but they were not used together (apparently that was extra paid option...) * pilots didn't know how the system worked or when it is active, so they were fighting against it

don't remember anything about overflow

[smugma]

It is useful to have a canonical example. Shared understanding of the problem, etc.

It’s related to why memes exist.

[cptskippy]

> Software must be pretty amazing and safe that we have to keep bringing up an example from the late 80s.

Or perhaps this failure settle the argument about whether or not physical lockouts were necessary and due to regulations you can no longer rely solely on software?

[thatguy0900]

More that software that is critical is written with some level of care, and then never touched again out of fear of breaking it. See all the stories of hospital machines running windows xp.

[MobiusHorizons]

That actually has a lot more to do with drivers not being provided by manufacturers for newer operating systems

[ilyt]

> but to paraphrase the designers eliminated the physical locking mechanism and had this attitude that "oh it's electronic so it won't wear out and fail".

And it would have been entirely fine if there was electronic safeguards put between code and rest of the device, just like some stop lights have it (if control board sends green signal to both directions it fails out and doesn't allow that). Defense in depth and all that

[angry_moose]

Covered really well in episode 39 of Causality, one of my favorite podcasts:

https://engineered.network/causality/episode-39-therac-25/

Each episode focuses on a root cause analysis of a major engineering disaster. Presented in a dry but engaging way - he's a good presenter, but none of the "Radio Lab" style nonsense.