[Retric]

You assume crash avoidance is the only reason a car would give up.

Self driving cars however have two goals, not hit anything AND got somewhere specific. It seems likely a car would disengage if it encountered a blocked road even if it did not hit anything. In that context we may see many cases where a car stops and 'gives up' which are in no way safety related and have no real safty risks.

While we don't nessisarily know the specifics, people have been testing these self driving cars for a while and yet crashing seems very rare even with unexpected handoffs.

[jacquesm]

I would hope in those situations it would just stop and not disengage.

It should remain in control until commanded to disengage. After all, blocked roads will be freed at some point and even if they don't it is the AI's problem to get itself out of any trouble that it got itself into. And even if a blocked road might seem to be free of safety risks that doesn't mean you can abandon the car there, you're supposed to stay in control of the vehicle until it is parked.

The only situation that I can compare disengaging with is when there is stopped traffic in the mountains and the snow moves in, that's one of those situations where you might be ordered by the authorities to abandon your vehicle and seek shelter (assuming this is possible and not less safe than staying with the car). Those situations can take many days to sort out afterwards. But in almost all other situations that normally would occur you should stay in and in control of your car, so I'd assume the same would go for an AI based system.

[Retric]

While that may be desirable if the car is operating alone in production, in testing just about anything it's not sure of should probably result in a disengagement. From my sensors are giving conflicting information with my map, to I failed to read that sign... they all should probably result in a hand off.

After that hand off someone can look at the test data and chose a better response, but caution when operating a deadly vehicle is causation is perfectly appropriate.

[jacquesm]

Agreed, but in that case it shouldn't be on public roads to begin with. If your software is so bad that the solution to complex situation is to throw up its hands and panic it has no place in traffic. After all, there are plenty of situations where inaction is just as dangerous or even more dangerous than action.

I feel pretty weird in the knowledge that people are operating vehicles with what to me comes across as barely out of beta software on roads shared with others.

[Retric]

Self driving cars safety track record during testing suggests these are already very well designed systems.

I also don't think you can get to self driving cars without testing them on public roads. So far it's been very safe and potentially it's going to save millions of lives so I don't have a problem with this.

Further, these cars are not simply turning off in traffic they are making errors known while continuing to drive as safely as possible. That's vastly safer than suppressing errors which generally results in people ignoring them.