[tqdm]

The only similarity between cooperations and AGI is that they are superhuman optimization processes that yield considerable risk and that are tricky to control. The difference is that in case of cooperations there still is a human element: vulnerability/fallibility, social reputation/integration, slowness/interpretability, empathy. The policies for mitigating the respective risks are very different.

[icc97]

He connects the corporation to the description of the Chinese Room. A corporation is a very big room full of people with human level intelligence under an artificial union acting as one.

Perhaps you can find other areas where that breaks down, but as far as reasonable guesses as to what will happen with AI in the near furture, this is the best I've heard so far.

[tqdm]

An AI that looks like a cooperation will likely do so for a very brief time before it can leverage computational resources far greater than than the equivalent of a group of cooperating humans plus simulation tools made from silicone. It is not an interesting comparison because it does not challenge the arguments that human potential is strongly limited by the size of the cortex and by the slowness of neurons and therefore of linguistic information transmission.

[codewithcheese]

That implies that "vulnerability/fallibility, social reputation/integration, slowness/interpretability, empathy" are not useful for decision making. Until we see intelligence comparable to human I don't think we can make that assumption.