[ItsDeathball]

I've thought something similar for a while, and it's been long enough that I think the problem is the failure mode. It might be that getting a car to recognize when it's encountered an edge case and needs to kick it to a human is about as challenging as just getting the car to come to the right conclusion and drive itself anyway. Intuitively we think of the problem as what the car needs to do, but the main problem is ascertaining the state of the road accurately. In the autonomous crashes we've seen from Tesla and Uber, it looks like the cars never even noticed enough to have an ambiguous condition they could deactivate autopilot from. They just had a completely erroneous perception of the outside world.

[wpietri]

And I think that's due to a "let's keep driving" bias. In Uber's crash, for example, it detected an object moving in the road, but didn't understand what it was. Any reasonable human driver would have quickly stopped until it was clear what was going on. But the Uber car just kept going at speed until it was sure that it was a pedestrian, at which point a collision was inevitable.

I think the right thing to do is to invert the bias. If the car isn't sure it understands what's going on and that it's safe to proceed, it should stop. In which case, development becomes a long series of teaching about rarer and rarer circumstances, so it needs less and less human input to understand the road.