[Bakkot]

I've never understood this logic, especially since you're deciding on the rule for all cars, not just your car. Shouldn't you prefer that the rule be to minimize the number of people harmed, since this is the option which (when taken over all cars) will minimize the likelihood of you being harmed?

[jack-r-abbit]

I actually put emphasis on "my" for a reason. I was talking about my robot driver. You should be free to instruct your robot driver how ever you wish.

[Bakkot]

Yeah, sure, but now suppose that you're going to decide the rule that everyone's car will follow. Are you still going to choose what amounts to defecting in the prisoner's dilemma?

Put it another way: this is one of those very rare occasion where collective cooperation (a la prisoner's dilemma) is possible. Shouldn't that... be the thing we want?

[jack-r-abbit]

I would not be deciding that rule for everyone's car. I would provide a checkbox (or what ever way there would be to customize a car's options). I'm a pretty big believer in user settings. I wish more software gave users more options on some of the stuff some random developer thought was best.

[Bakkot]

The thing is, in the absence of collective action people choose to defect in prisoner's dilemmas. Giving people the option results in everyone defecting. Having it be preset results in everyone cooperating. Everyone cooperating is a better outcome for everyone.

[krapp]

Isn't the whole premise behind autonomous cars that humans shouldn't be able to decide how they operate, beyond giving them a destination?

[jack-r-abbit]

Will autonomous cars prevent me from choosing the radio station or the cabin temperature? No. Why should they prevent me from choosing if I live or die?

[krapp]

Autonomous cars are supposed to be more intelligent, and capable of making better decisions than a human driver, based on access to more information and having more accurate controls. The entire argument about autonomous cars improving safety is predicated on the assumption that human drivers, in general, are simply not capable of driving safely.

You being able to make a decision about the way your autonomous vehicle behaves which could result in a greater risk to other drivers is no different than the risk you present anyway just by having access to a steering wheel, and making the equivalent decision directly. Changing the radio isn't going to kill anybody, but letting you decide the ethics of your car just might. What happens when two cars with conflicting driver-set ethical subroutines come into contact? Do they debate? It doesn't work unless there's a single standard.

[jack-r-abbit]

An autonomous car is still going to have a finite number of actions it can take. Using the info it gathers on the fly, it will traverse a pre-programmed decision tree to arrive at the action(s) it should take. Being a computer, of course it will do this faster and use a lot more information than a human would. This is going to result in "driving safely."

I'm not saying we should be able to rewrite the code. But a single user option like "Should I sacrifice your life to save another? [Yes/No]" just means the decision tree has another branch point that still leads to the same set of possible actions. It can still make better decisions about driving than the human driver because it still has the access to more information and more accurate controls. Deciding to crash me into a tunnel wall to save a child is not really about "driving safely."

[krapp]

Logically that makes sense, but practically, no car manufacturer is ever going to add a "who would you rather I kill" option to their UI.