[strbean]

This is all speculation based on the development of AGI. If we achieve AGI, yes, it would enable humanity to have a Star Trek-style fully automated space communism utopia. However, that hinges entirely on access to AGI and the fruits of it's productivity being distributed across the population, instead of hoarded by a small group.

If AGI is developed and kept closely guarded, whoever has it will have essentially limitless productivity, and quickly concentrate all wealth and power. They wouldn't even need to engage with markets, they could simply build overwhelming autonomous military power.

[visarga]

Here you are making a mistake - AGI by its nature should know all there is to know, and yet need to make progress by searching for new approaches and discoveries. That doesn't happen all at once, it works field by field, and discoveries actually come from experimental validation. There is no "secretly developing AGI" to "quickly concentrate all wealth and power".

Like bitcoin, you are basically saying someone could outcompute humanity and own the ledger. But in reality the combined research power of humanity, which is necessary for AGI to advance further, is much deeper than any one entity could achieve in isolation. Research is a social process.

[Nevermark]

> the combined research power of humanity, which is necessary for AGI to advance further, is much deeper than any one entity

AGI won't need people to advance further. That is pretty much the functional definition.

But the bar is even lower. Just as today's rich keep increasing the economic distance between themselves and the poor, with the middle and creative classes already feeling that gravity, so will the AGI's - even if they stalled out only as smart as we are, but cheaper in inference mode. (A limitation that is highly unlikely.)

No doubt humans will facilitate AGI activity in our economy and real world for practical reasons for a little while. But at some point, they won't need us physically or socially either.

[visarga]

> AGI won't need people to advance further. That is pretty much the functional definition.

No, AI can ideate as well as any human, but that is not sufficient. Scientists ideate and test. Engineers design and test. There's always a validation stage, where ideas meet the bottleneck of reality.

Are you saying scientists without labs and tools to run experiments on could do science? It's all in the brain or GPU? That is so naive.