[tim333]

I'm not sure it's true they must be programmed to kill. In general they will be programmed to avoid killing anyone. I imagine in situations that they do kill people it will be because the systems failed rather than by calculation.

[cpncrunch]

Their hypothetical situation seems a little dubious, and it seems unlikely a self-driving car would ever allow itself to get into a situation like that in the first place.

[sokoloff]

Because no self-driving car will ever have pedestrians dart out in front of it?

[mcv]

In areas where this is likely to happen, cars don't go that fast and should have plenty of time to stop. If pedestrians are actively trying to commit suicide and the only way to prevent that is to kill the driver or an innocent bystander, then hitting the suicidal pedestrians is preferable.

More generally, I think the car should try to stay on the road and it its lane whenever possible, and only leave that space when it's absolutely safe to do so. People on the street have chosen to interact with cars, people on the sidewalk haven't. And the best way to avoid an accident is usually to brake, and not to swerve into someone else or into a wall.

[sokoloff]

The question the article raises is not "should an autonomous vehicle ever use its brakes to avoid an accident?" but rather "in a situation where brakes are insufficient and the controller of the vehicle needs to make a decision about how to weigh the risks to various people, how should those decisions be made?"

[mcv]

That doesn't change the fact that this is a situation that should not occur. When it happens, someone fucked up. Maybe pedestrians are throwing themselves into oncoming traffic, or the car has already made some pretty terrible mistakes.

[cpncrunch]

At 30mph? Or are we assuming 10 pedestrians are somehow running out onto a major highway? I'm struggling to see how this is a valid scenario. If you can think of one, please share...

[sokoloff]

Two kids run out into the street chasing a ball. Kill the kids or the occupant?

Even one kid runs out into the street. Hit the kid straight ahead or risk the occupant(s) by crashing the car or turning the car into an unknown area?

Car is about to be struck by a red light/stop light runner from the left. Swerve off the road to prevent it (into an area of "unknown contents, possibly including pedestrians") or just take the hit without evasive maneuvering?

There were multiple scenarios researched via MTurk in the story, with variable numbers of pedestrians.

[cpncrunch]

>Two kids run out into the street chasing a ball. Kill the kids or the occupant?

As I mentioned in my previous post, how exactly is that going to kill the driver at 30mph? There isn't going to be a continuous brick wall at the side of the road, otherwise where the heck did the kids come from?!

I think the article is a little overhyped. Certainly there are cases where the car will need to take evasive action, but it's unlikely it will have to decide who to kill.

[sokoloff]

Agreed that even a continuous brick wall would be relatively safe for an approximately tangent collision at 30 mph.

The concern would come from a break in a guardrail or a series of concrete columns/bollards with a pedestrian crossing, where the car could strike an abutment. (Think of "Jersey barriers" lined up in a construction zone through which a zebra crossing lies. That's an easy abutment scenario and striking a line of Jersey barriers end-on at 30mph provides a deceleration approximately equal to a 30 mph head-on collision with an opposite direction car of equal mass also traveling at 30 mph. That's going to be a massive shunt.)

I agree there's an overhyping in the title, "programmed to kill", but the decision of "cause a near-certain harm to a pedestrian" vs "cause the car to leave the known roadway and suffer/inflict unknown and unknowable harm" is a very legitimate concern.

(I worked as an intern for Daimler on a completely autonomous bus. Our only fallback was "mash the red E-stop button [shutting off the computer servos] and manually take over", so we literally never let the vehicle exceed what the human safety driver was comfortable with, but eventually that's going to go away for fully autonomous vehicles.)