[Razengan]

The first priority should always be your occupants. Hardcoded.

Otherwise, can you imagine the first incident in which some passengers died because the code prioritized the life of someone else? Would anyone want to buy an autocar after that? "YOUR CAR COULD CHOOSE TO KILL YOU", the headlines will say.

The second priority should be the safety of pedestrians who are off the road.

Until we have computational power at the level of divine prescience, I don't think you can do much else in scenarios like the one you described, other than have a chain of basic priorities that falls back on "stop everything."

Eventually though, powerful-enough AI may be able to simulate all outcomes down to a very granular level, such as the difference between outright killing someone or just injuring their limbs to the point of crippling them for life.

Many factors will need to be considered then; who appears to be the weakest individual among the unavoidable targets? Are there any acquaintances of the owners among them? Is there a hospital nearby that could tend to their injuries? Does it look like a suicide attempt?

[randomThoughts9]

The government might and probably will change this rule.

I can imagine that after the first couple of accidents where the algorithm decides to kill N pedestrians to save 1 driver, the government could switch to a "utilitarian" point of view - minimize the number of lost lives - and will enforce this by law.

[thaumasiotes]

> the government could switch to a "utilitarian" point of view - minimize the number of lost lives

There's nothing "utilitarian" about this, unless you assume that:

(a) All people value their own lives equally

(b) Nobody places any value on anyone's life but their own.

(Side note: we know that both of those don't even come close to being true.)

Without those assumptions, a utilitarian point of view will show some pretty obvious biases against just killing the fewest people -- among other things, it will try to kill older people rather than younger people, and to kill people with the smallest number of relatives (who would be sad about a death in their family).

[ewindisch]

Right. We had an extensive discussion about this at BarCamp Philly this year. The discussion was on machine ethics and we spoke largely about these scenarios of driverless vehicles.

I made the argument that machine ethics aren't a real thing. Companies will do whatever they can to avoid litigation, then the government will step in with regulations as the closest approximation we'll get to ethical behavior.

[TeMPOraL]

I hope one day people will actually grow to appreciate that viewpoint and care about others too. Personally, I find the concept of a car that would consciously chose to save you at the expense of 5 pedestrians to be morally wrong.