[erikpukinskis]

There is a corollary prediction: AIs will disagree.

Too many people imagine AIs are somehow "perfectly rational" and therefore will never disagree, they'll just tap into a unified decision making engine and automatically integrate all of their knowledge.

In order to integrate knowledge you need more knowledge. And in order to integrate that you need more still. It's turtles all the way down. If anything, AIs are at a disadvantage to humans, because we at least share a body, an emotional palette, and to some extent a cultural timeline. AIs have nothing to ground the beliefs of the other in.

[Felz]

Have you heard of Aumann's agreement theorem? https://en.wikipedia.org/wiki/Aumann's_agreement_theorem

Theoretically it means that two AI should be able to reach a consensus pretty quickly without needing to share a lot of information.

Unfortunately you need "honest, rational Bayesian agents with common priors" for this to work. Given that humans rarely agree with each other, it's interesting to think about where we fall short of that criteria.

[digi_owl]

If AI is to agree they need all the facts. And as we keep adding facts almost daily, there is simply no chance that the AI will have all the facts.

We like to think of logic as 1+1=2. But logic is more like A+B-C*D/E=F. Where any number of those being either completely unknown (we sometimes do not even know we are missing them), or some wide range of potential values.

[erikpukinskis]

You need a third model to map facts from one model into facts in another model. Doesn't really solve the problem.