[ghaff]

I'm not sure it's a trolley problem so much as an automated system that is statistically safer than manual operation but will "randomly" kill people on a regular basis. Which is a legislative/liability problem because outside of possibly rare drug side effects, we don't normally accept consumer-facing products, even if used and maintained properly, randomly killing people--and just shrug our shoulders because stuff happens.

[bumby]

I think the OP was referencing the trolley problem because autonomous driving AI is often regarded as a utilitarian problem. In other words, "intent" does not matter, only consequences. In that regard, any ML cost function is purely focused on consequences. In the context of minimizing deaths it wouldn't matter if the software deliberately chose to kill one person, if the cost (presumably total deaths) was minimized. Where I think it gets sticky is that our society does factor in intent (see the various degrees of homicide) and if we're thinking in terms of pure utilitarianism, I'm not sure how this plays into the liability of autonomous software.