[lampiaio]

The ethical/moral conclusion(s) AGI may arrive will most likely put said "ethical pluralism" to the test. The pluralism we claim to have may be only a small subset of what's really philosophically possible. Will we still claim to be "plural" when an all-knowing AGI uncontradictably concludes something that is anathema to all humans? We may discover we only like to think we embrace pluralism. AGI may show us that even our most opposing schools of thought are simply shades of a same color -- and may do so by forcing a whole spectrum of never-seen-before colors upon us. I say humanity is not emotionally ready for what could happen. We are not prepared for the plural conclusions AGI may arrive.

[JumpCrisscross]

> we are not prepared for the plural conclusions AGI may arrive

Plenty of criminals believe they acted ethically. We don’t set the justice system on fire every time someone credibly claims their crimes were justified.

[lampiaio]

That is very much my point. Maybe such justification attempts required a reasoning capability much beyond that of a human person. And curiously, there are also plenty of stories where we find the perpetrator of a crime to be justified in what they did. We sure are not ready for a greater intelligence saying we are wrong about things we are adamant about.

[JumpCrisscross]

> sure are not ready for a greater intelligence saying we are wrong about things we are adamant about

I almost hope you’re right, because it suggests a greater role for rational debate. In reality, people ignore arguments they don’t like. To the extent a greater intelligence realised this, the advantage would be in manipulating us with better propaganda, not penning a treatise.