[jschwartzi]

They are absolutely negligent for doing so. I wouldn't have made that decision, because in a safety-critical system it's ALWAYS preferable to have a backup system, especially when the system you're working with is unproven software that you don't fully understand. It doesn't matter that the old system would have interfered with the new system, and in fact if the two systems did interfere it would behoove you to understand why. The decision speaks volumes about their engineering culture leading up to this incident.

You can't just call something a safety system. You have to prove that it is a safety system by testing it, which is something that Uber hadn't done before they disabled the existing system.

[nmca]

So I agree that the system was insufficiently tested to be operating on public roads and endangering lives, and that lack of testing was negligent; I also agree with your comment about their engineering culture.

But those two points seem distinct from the idea of disabling a system in order to have a better understanding of what's going on. Suppose they had built a car that was safe, conditioned on the presence a black box system that they likely didn't have access to the internals of - would this be satisfactory?

[mannykannot]

Even with the red herring about the safety overrides being a black box, yes, it would be satisfactory - see my other post. Not only would the triggering of the safety override generally indicate a failure of the autonomous system, the use of failsafe overrides to catch corner cases should be a feature of the final system.

If Uber could demonstrate, through the analysis of a statistically significant number of events, that its system was actually safer without the car manufacturer's override (e.g. if all the events were false positives), then it would be appropriate to disable it at that time. That's how you do it.

[nmca]

Replying to your other comment here as well - the inclusion of conceptually simple safety mechanisms is important (eg I agree), as is the broader scheme of including both hardware and algorithmic redundancy to improve safety. I also agree that "live" initial testing of such safeguards is inappropriate, and as above Uber clearly failed to do appropriate testing.

However you describe the (potential) black box nature of the existing system as a red herring -> to be honest, this is what I'm most interested in. My opinion is that including a black box component into a saftey-critical system would be inappropriate. Do you disagree with that? If your answer is "probe it until it's no longer a black box and then include it", would you not consider that to be overall semantically equivalent to "don't include a black box"?

[mannykannot]

It is a red herring because:

Firstly, it assumes Volvo is not sharing the parameters of the system. It seems unlikely that Uber is installing an automated driving system into these cars without the cooperation of Volvo, especially with the agreement to ultimately get 24,000 autonomous-system-equipped cars from them.

Secondly, if Uber could instead determine what it wants to know about the parameters by testing, then the question is irrelevant, as are the semantics.

Thirdly, it is presumably safe for humans to be driving cars without knowing the exact parameters, and so should not present any particular problem for the autonomous system - if the emergency brakes are triggered, it is likely to be a situation in which it is the right thing to happen, and possibly a result of the autonomous system failing. Just as for human drivers, an autonomous system is expected to usually stay within the parameters of the emergency system, without reference to those parameters, just by driving correctly. For example, if the emergency brakes come on to stop the car from hitting a pedestrian because the autonomous system failed to correctly identify the danger, what difference would it have made if the system knew the exact parameters of the emergency braking system?

Lastly, the road is an environment with a lot of uncertainty and unpredictability. If the system is so fragile that the tiny amount of uncertainty of not knowing the exact parameters of the automatic braking system raises safety concerns, then it is nowhere near being a system with the flexibility to drive safely.

It is possible that a competent autonomous driving system might supplant the current emergency braking system, in which case the way to proceed is to demonstrate it in the way I outlined in the last paragraph of my previous post.

[nmca]

Thanks for answering in so much detail - I think the last two points make a compelling case for not disabling the system, even in the true black box case, and the first two are very compelling in the real world, even if they don't apply to the thought experiment of an actual black box. You've broadly changed my mind on this issue :)

[mannykannot]

I should have said that your concern is valid where two systems might issue conflicting commands that could create a dangerous situation, it is just that I don't see it likely in this particular combination of systems.

[jschwartzi]

It's perfectly appropriate when both systems are intended to enhance safety. It does not matter how the internals work, just that the system enhances the safety of your solution. It could be a webcam beaming images back to a bunch of Mechanical Turk operators and it wouldn't matter as long as it was proven to work.

It's not like Software of Unknown Provenance where it's running in the same execution environment and you don't have any control. This was a completely separate system with independent sensors that was marketed to stop the car in the exact situation the car failed to stop in. Disabling it was foolhardy.