[asynchronous13]

There is a long list of entirely preventable human-caused accidents. Is there a reason pilot-caused crashes are less scary for you? Computer caused accidents will be fixed and won't happen again. Human-caused accidents will keep happening as long as experience is valuable.

    Aeroflot Flight 593 - pilot let his son fly the plane, 63 dead
    Germanwings Flight 9525 - (possibly suicidal) pilot deliberately crashed , 144 dead
    Air France Flight 447 - pilot caused airplane to stall, 228 dead
    Aero Flight 311 - both pilots got drunk, 25 dead

and this is just a random selection, there are long long lists of human-caused aviation accidents.

[Simulacra]

Don't forget Colgan Air Flight 3407 where the pilot was either too sleepy, or simply didn't know what he was doing.

[gonehome]

I'm not a luddite, if the software is ready and safer than people then I'd be okay with it.

There's a history of software not being ready while people pretend it is and then it kills people (Therac-25).

I'm just skeptical that we'll know when it's actually safe.

[DaiPlusPlus]

Therac-25 wasn’t a “it’s not ready yet!”-type issue. It wasn’t an expected or anticipated failure-mode - it only became a (literal) textbook case-study after people died and the industry has learned and improved as a consequence.

[gonehome]

They ignored repeated failures and evidence of malfunction by saying it was “impossible” that it could be failing in that way.

Unexpected failure modes are the issue. The Boeing 737 max 8 failure being tied to one sensor would suggest the industry has not fully learned the lesson.

[DaiPlusPlus]

My understanding is that it was a UX issue - the "malfunctioning" was the system working as-directed by the user, but the UX was horrible for informing the user what they were doing.

[gonehome]

It was a lot worse than that: http://sunnyday.mit.edu/papers/therac.pdf

That paper is long, but does a great job of giving the context and going deep on the details.

[DaiPlusPlus]

Thank you!