[JoachimSchipper]

Eliezer's answer seems to be "do no harm, even when this seems to be beneficial, as your judgment is likely to be wrong".

For instance, quite a few dictators seize power and suppress dissent because they honestly believe that's the best for the people. It's all too easy to convince oneself that something that is convenient for oneself but harmful to another is, ultimately, "for the greater good".

Of course, exceptions to the "do no harm" rule do exist. However, the probability of the current case being an exception may be vanishingly small, even if you are convinced it is an exception. If this probability is indeed very small, the "do no harm" rule produces a better expected outcome than a "pragmatic/utilitarian" point of view for a human (imperfect) agent.

In the philosopher's "100% sure" case, a true mathematical weighting can be made and (true) utilitarism wins; but no human can ever be that sure.