[aclissold]

It’s interesting to observe that, if this is indeed caused by a failure in the stall prevention system (and I say “if” since it’s certainly too soon to draw conclusions), the media and discussion around it seems to gravitate towards disabling the system or avoiding the aircraft entirely.

But when we think forward to the inevitable autonomous vehicle accidents that will occur, the conversation turns to how many lives they’ll have saved, and how much safer they’ll be.

Is there a known psychological phenomenon for “negative hindsight, positive foresight” that I can go learn more about?

[Xixi]

The problem is that you can't disable that system: the aircraft is not certifiable without, that is to say the risk of stalling is considered too high without that system. Because that system may have caused two planes to crash doesn't mean that it didn't prevent the ~350 other 737-MAX flying from stalling.

So if the MCAS is indeed the cause of that crash, I expect fleet grounding until the system is fixed. Assuming it is fixable, which I think it is, but might require more than just a software fix.

[jimktrains2]

> But when we think forward to the inevitable autonomous vehicle accidents that will occur, the conversation turns to how many lives they’ll have saved, and how much safer they’ll be.

Driving to the airport is orders of magnitude less safe than flying. If a malfunctioning automatic system is decreasing the reliability of flying, that's a huge problem. However, I'd wager that even an autonomous car that's only just able to pass a driver's exam would be significantly safer than a human driver because it would _follow the rules_ and _not be distracted_. Even if that system isn't perfect, it's probably still better than an experienced driver.

[dwighttk]

Except we test humans with tests designed to show that a human who is capable of generalizing knowledge can do a few things that prove they can do more things than that. If the AI isn’t capable of that then just being able to pass the drivers test might create a very incapable AI

[salawat]

This is what scares me about the push toward fly/drive-by-wire.

At some point your designs start breaking into envelopes where the machine cannot be considered safe once the automated systems fail, making your pilot/highly trained human being powerless in the face of catastrophic system failure.

An uncontrollable tool is not a tool, but a coffin waiting to happen. I don't think any type of "routine" transit system should be designed in a manner such that it so thoroughly overwhelms a human crew's workload that it should be so dependent on automation that it cannot be certified otherwise.

To reword: if it can't be safely flown with the computers off, it probably should not be a design we allow for people transport. Markets be damned. When your margins include human lives lost, economy needs to stop being your primary optimization. Dollars should only be important after you stop being a corpse factory.

[aclissold]

Interesting point.

I guess whether it’s human error at the hands of the pilot/driver, or human error at the hands of the engineer/designer, we can never fully remove it from the equation.

Should we just give up? Seems the best we can do is try and mitigate risk, and automated systems condense this risk down into fewer points of failure (i.e., there are less engineers than users!).

[sam_lowry_]

The difference is, the pilot is one person, and his life is in danger, so he is dead serious. Designers/engineers work in teams, supervised by managers and driven by market. Responsibility is diluted to the point that evereyone feels they did nothing wrong, although the results are catastrophic.

[sn41]

I am sure that an autonomous system that does not have a manual override will have the same pushback against it.