It's honestly like a black mirror episode. "I have to have maintain a mental model of how my black box ML car AI will act so I don't die or kill other road users... and it can be updated daily."
Autopilot in aircraft have a similar problem, which led to the Air France Flight 447 crash in 2009. The airspeed indicators froze over, the autopilot switched from "normal law" to "alternate law", and the pilots failed to understand that the autopilot's protection from stalling no longer applied. They did not have enough training (or were too confused) to fly without the help of automated aids and flew the plane right into the ocean.
Boeing's planes were known for having fewer of these automated aids and thus required more skill to fly the plane, but the 747-MAX debacle threw cold water on that idea.
Of course, (unlike some commenters in this thread) the air travel industry thankfully had the good sense to look at autopilot on balance rather than fixating on a single fatal accident (or in the case of Tesla's Autopilot, the mere possibility of a fatal accident). The salient question isn't whether or not an autopilot allows for any fatalities, but rather whether or not it reduces fatalities relative to manual piloting/driving.
Yep, autopilot is a good helper but a bad master. In the air there's a lot more space and fewer objects to avoid than on the roads, yet nobody has managed to build a perfect autopilot that could really replace pilots in every situation. I remain skeptical that we'll ever see one in any dimension, let it be air, sea or land.
> "I have to have maintain a mental model of how my black box ML car AI will act so I don't die or kill other road users... and it can be updated daily."
Not defending Tesla here, but this is exactly how I think of other drivers on the road, whether I'm driving, cycling or walking.
Yes, we are literally trained for that - to keep distance with cars ahead of us, expect the unexpected and so on. We are also told not to drive cars without proper maintaince (and inforced by law in most places) so our cars are reliable.
I wonder if future driver training will factor in spotting problems with self-driving or related functions.
Boeing's planes were known for having fewer of these automated aids and thus required more skill to fly the plane, but the 747-MAX debacle threw cold water on that idea.