[qz_kb]

My main gripe about AGI is that everyone assumes a general intelligence will somehow be able to self optimize towards a more and more improved state never reaching a plateau. I think it's much more likely that the optimization landscape when searching for "higher intelligence" is full of local optima and does not have these "singularity" style ramps towards infinite intelligence that any self optimizing system could just discover and ride toward infinity.

There are millions(?) of human researchers (and orders of magnitude more computers) doing gradient free optimization through their research in this direction, and the progress is painfully slow, I know because I'm one of them. There are billions of years of optimization (evolution) towards this goal, and a total of (1) species has achieved any kind of notable intelligence. We are collectively giant parallelized optimization.

We already have "AGI" orders of magnitude more capable than any single human in the form of billions of people networked through the internet searching for fulfillment, money, power, fame, etc. for the next big discovery or supporting this effort by providing everything the entire global "machine" needs to run. The idea one little box running the right program can have access to the energy to beat this effort and exponentially improve things seems laughable in comparison.

The global "AGI" formed by all of us, the internet, and computers, is more likely to destroy society in the next 20 years in some catastrophic event than some paperclip machine.

[AgentME]

If we make an AGI at anything like human level, then even without any real progress past that, it will become superhuman just by removing biological limits (like being able to duplicate itself instead of having to convince and train other humans as we do), and groups of AGI can work together even easier than groups of humans. I think a lot of people describing the "singularity" are imagining scenarios like this too instead of just the idea of a single computer-sized AGI self-optimizing its algorithms to infinite intelligence.

I'm not sure whether I expect AGI to happen within some centuries but it's not ruled out, it's vaguely plausible, and the consequences would be extremely high so I think it's very useful for people to think about.

[qz_kb]

"removing biological limits" is a big handwave. Biological hardware is tuned exceptionally well for intelligence, and we have a massive amount of it distributed worldwide. We already remove biological limits by building tools, computers, powerplants etc. Duplicating one AGI unit now doubles the amount of energy and resources it needs, and also creates a requirement that the system can somehow produce more of itself and all the supporting infrastructure it requires. It's not just a matter of being "intelligent" (im not even sure what "intelligent" means in this context either) - it will need to have power to act in the world, and not get sidetracked over-weighting any one of the multiple competing optimizations in complicated real world problem landscapes it will need to navigate to scale.

[AgentME]

It takes decades of training to make new humans useful enough to participate in many subjects of research. Having AGI researchers means being able to duplicate any expert researcher at any time. Getting more computing equipment to run more AGIs costs money but so does training a human for decades. Even if the AGI is more expensive for some time, there will surely be research subjects that today have money to spare and not enough expert humans that will be able to explode in expert manpower through AGI.

>it will need to have power to act in the world

- Human researchers would bring AGI researchers into their teams and empower them to get stuff done.

- AGIs would be competent at making money through intellectual/remote work and have money to use to act in the world and get what they want.

>and not get sidetracked over-weighting any one of the multiple competing optimizations in complicated real world problem landscapes it will need to navigate to scale.

Humans face this obstacle too.