[testcase_delta]

To your first point, I'm not convinced that's true. People augment their lives in lots of ways that don't seem to reduce safety. A few examples off the top of my head: simple "dumb" cruise control hasn't lead to more accidents. Parachutes have auto-deploy features if the cord isn't pulled by a certain height. Scuba divers use dive computers that basically eliminate the need to learn dive tables (and beep at you when you're doing something dumb). Apparently passenger jets are highly automated (I'm out of my depth on that one). These are all on the spectrum towards automation and have only been helpful. Do you think the problem occurs as you approach 100%? Like an uncanny valley in the 99 to 99.99% range?

[erobbins]

The thing with other activities (diving, flying are great examples) is that when a problem occurs, you generally have minutes to analyze what's happening and decide on a solution. If my dive computer goes on the fritz, I can decide to immediately start an ascent, or go off physical dive table, or make an extra safety stop at say 20 feet just to be sure.

When you're going 45 in a curve driving along PCH, and a sudden fog bank obscures your cameras and LIDAR and the computer says "your controls, good luck!" you have maybe 2 seconds to react, if you're lucky. It might be a lot less.

Humans make really dumb decisions sometimes, but we are also outstandingly capable of reacting to novelty.

[K0SM0S]

Not OP but to your last question I think there's proven danger in the "too familiar, too easy" zone, i.e. most car accidents for instance tend to happen in places you know pretty well — hence why you may get surprised when things happen out of the ordinary.

Whether it's an illusion of safety, a letdown of attention, the general idea is that humans should never trust that things will go well when there's real probability that they don't. I think it's not in the amount of automation, as you explained well, but rather in focusing users on the critical parts that they should watch out for — and there clearly automating helps us remove the unimportant from the equation, and also make us more responsive, more accurate for the important parts. But it's a lot of great UX, and that's one field where e.g. the military is usually great but commercial companies are abysmal if they can get away with it (read: sell enough to justify not spending a dime on more quality). That's worrying when security is involved, but it hasn't proven a moral or ethical problem for most industries absent of regulation (forced ethics, ha!), so... I think there's valid concern by OP.

As for passenger jets, the Airbus A320 (late 1980s) was the first commercial plane to have a "full" autopilot; all systems were electrical¹ (manoeuvering, thrust control, etc) which allowed the computer to integrate and manage it all. :)

It was tested a number of times by pilots for fun, from taking off to landing entirely on autopilot — ofc they're standing right there ready to take over if anything goes wrong but I've seen it first hand many times. We're talking commercial flights with passengers, it's 100% safe and actually quite "smooth" because the computer is so accurate.

Honestly, the problem is much, much, much easier for planes: a good GPS and it becomes quite the closed problem, and obviously 100% of autopiloted planes are simultaneously piloted by real humans... ready to take over. Yet a plane could technically land itself just fine if pilots were incapacitated, it really could. I suspect it did more than we know for many reasons. And when flying by instrument (means you see s__t), an autopilot is basically just a computer doing what a human would do slower by reading the same data (and maybe cross-checking with physical/manual instruments, but an autopilot doing the grunt work of stick-holding gives you more time to double, triple-check everything incidentally).

_____

[1]: Note that all systems are also doubled (even tripled) with mechanical (hydraulic etc) failovers, because obviously you can lose electricity in catastrophic situations, hence why it always seemed crazy to me that a planed requires software to fly properly instead of plain old good physics and mechanics).