[HALtheWise]

The trolley problem is usually phrased with a "bystander" making the decision, but is simplified to the point of ignoring most of what is important about being a bystander, in ways that makes people's moral intuitions look unnecessarily silly precisely because it's an artificially simplified scenario. In particular, typically bystanders in the real world are both numerous and uninformed. As a result, assuming that each bystander has a randomly biased estimate of the truth, if everyone followed the simple logic of pulling the lever iff they believe doing so will net save lives, the lever is likely to be pulled far more often than it should be, because the person with the most extreme belief will conclude that doing so is worth it. In the real world, it's typically the case that "action" is much more difficult to reverse than "inaction".

There are other thought experiments that elucidate this more clearly, like a soldier deciding whether now is the right time to fire the first shot to start a battle.

People's moral intuitions have heuristics that attempt to deal with this ("bystander effect") although they definitely can be poorly-calibrated in the real world, and are almost always badly calibrated in artificial thought experiments because that's not the environment they were designed for.