> we are not prepared for the plural conclusions AGI may arrive
Plenty of criminals believe they acted ethically. We don’t set the justice system on fire every time someone credibly claims their crimes were justified.
That is very much my point. Maybe such justification attempts required a reasoning capability much beyond that of a human person. And curiously, there are also plenty of stories where we find the perpetrator of a crime to be justified in what they did. We sure are not ready for a greater intelligence saying we are wrong about things we are adamant about.
> sure are not ready for a greater intelligence saying we are wrong about things we are adamant about
I almost hope you’re right, because it suggests a greater role for rational debate. In reality, people ignore arguments they don’t like. To the extent a greater intelligence realised this, the advantage would be in manipulating us with better propaganda, not penning a treatise.