[jacquesm]

Interesting observation. And yet, societies have many mechanisms to reduce the amount of ethical pluralism such as laws, conventions & customs, peer pressure, religions and so on. It seems as though we will tolerate some ethical pluralism but not too much of it. There is this 'bandwidth' of acceptable behavior and if you go too far out of it bad stuff will happen to you: you get ostracized, put in jail, a psychiatric institution or a re-education camp, and in extreme cases you're simply murdered.

We have lots of ways to deal with people that exhibit too much 'ethical pluralism'.

[JumpCrisscross]

> societies have many mechanisms to reduce the amount of ethical pluralism such as laws, conventions & customs, peer pressure, religions and so on

Constrain, yes. The same way we would seek to constrain a paperclip-maximising LLM.

[mrob]

Any AI smart enough to become a classical paperclip maximizer is smart enough to hide its abilities and intentions until humans no longer have the ability to constrain it.

[pas]

That means we need to setup hidden societies with enormous capabilities to strike at potential runaway AGIs. Free Masons with EMPs!