[AgentME]

If we make an AGI at anything like human level, then even without any real progress past that, it will become superhuman just by removing biological limits (like being able to duplicate itself instead of having to convince and train other humans as we do), and groups of AGI can work together even easier than groups of humans. I think a lot of people describing the "singularity" are imagining scenarios like this too instead of just the idea of a single computer-sized AGI self-optimizing its algorithms to infinite intelligence.

I'm not sure whether I expect AGI to happen within some centuries but it's not ruled out, it's vaguely plausible, and the consequences would be extremely high so I think it's very useful for people to think about.

[qz_kb]

"removing biological limits" is a big handwave. Biological hardware is tuned exceptionally well for intelligence, and we have a massive amount of it distributed worldwide. We already remove biological limits by building tools, computers, powerplants etc. Duplicating one AGI unit now doubles the amount of energy and resources it needs, and also creates a requirement that the system can somehow produce more of itself and all the supporting infrastructure it requires. It's not just a matter of being "intelligent" (im not even sure what "intelligent" means in this context either) - it will need to have power to act in the world, and not get sidetracked over-weighting any one of the multiple competing optimizations in complicated real world problem landscapes it will need to navigate to scale.

[AgentME]

It takes decades of training to make new humans useful enough to participate in many subjects of research. Having AGI researchers means being able to duplicate any expert researcher at any time. Getting more computing equipment to run more AGIs costs money but so does training a human for decades. Even if the AGI is more expensive for some time, there will surely be research subjects that today have money to spare and not enough expert humans that will be able to explode in expert manpower through AGI.

>it will need to have power to act in the world

- Human researchers would bring AGI researchers into their teams and empower them to get stuff done.

- AGIs would be competent at making money through intellectual/remote work and have money to use to act in the world and get what they want.

>and not get sidetracked over-weighting any one of the multiple competing optimizations in complicated real world problem landscapes it will need to navigate to scale.

Humans face this obstacle too.