[PeterisP]

The point seems to be that is you limit your policy to something that can be reasonably be encoded as an algorithm and implemented as a program then you're by definition limited to write faulty policies that can't be flexible enough and will harm people.

I.e., that if you're choosing to use an algorithm for your policy, then this means that you will write harmful policies, and the choice to use an algorithm at all, any algorithm, is morally flawed and should be vilified - to motivate you and others to write policies that include appropriate flexibility, arbitration, human evaluation, overrides and thus can't be implemented as a scalable algorithm/program. Well, not unless we get superhuman general AI systems.