[bumby]

Managing those edge cases effectively is extremely important in safety critical software. Addressing these is often what separates quality critical software from low-quality. Would you want an aircraft software engineer or nuclear power plant engineer to disregard low probability events? It worries me when I get the impression the SV mindset of glorifying “moving fast and breaking things” infiltrating safety critical applications, particularly those that impact the general public.

>Second, the solution is simple anyway: favor the passengers of the current car.

I don’t think this is a given. Would the ethical software, for instance, drive a car through a crowd in order to save a lone passenger? Would you be okay with a human driver being absolved of responsibility for the same choice? My intuition is most would not, because we recognize there is an obligation to others as part of the social contract. It seems a naive oversimplification to not extend the same obligations to software that makes the choices in our place.

[MattRix]

How is the car saving the passenger by driving through the crowd? That seems like an entirely imaginary scenario. In almost any real world scenario the solution is going to be to apply the brakes and come to a stop.

[bumby]

It is an imaginary scenario. But so is the scenario where they can always apply the brakes and come to a stop.

What if there is a cement truck tailgaiting and they can't hard brake? What if there are jersey barriers constraining the maneuverability? Either of those would risk the passenger more than running into pedestrians.

Performing failure-mode-effects-analysis on software is largely an exercise in creating imaginary scenarios and then effectively mitigating the risks to within an acceptable level. I agree, in most real-world scenarios the tough problems can be mitigated largely by "slow down and re-assess". But that's not what safety-critical software risk mitigation is always about. You have to mitigate low-probability events as well, especially in a domain with governmental regulation. According to the NTSB report, the Uber accident that killed a pedestrian applied the "slow down and re-assess" scenario that attributed to the death. The software was misclassifying enough that an artificial delay was put in to avoid (I assume) nuisance braking. So in the real-world, they didn't want to always apply the "just brake if in trouble" strategy because of trade-offs to the passengers and it didn't work out well.