[visarga]

This jump from "AI assistance" to "millions or billions of destitute people" is unjustified. Yes, AI can assist, but it doesn't replace human presence as it is now, in any field. Instead it creates demand for improvements across the board and creates more work for us.

When we went from horses to cars, we increased the volume of shipments and percentually more people work in transportation now than in 1910.

Another example - programming has been automating itself for 60 years. Each new language, framework or library, each project on Github makes future work more efficient. Yet we have seen an increase, not a decrease, in development jobs, and good salaries.

I would say humans are the critical ingredient AI needs to be effective, at least for now. And in the far future where AI can work without assistance, then it just empowers everyone to not need to work. We can use AI directly for our own benefit, automating self reliance for people. Unlike UBI, you can copy an AI and give it to everyone for free.

In both scenarios: weak AI making room for jobs, or strong AI making work not necessary, it turns out ok. But have more faith in our insatiable greed, we won't run out of work before we run out of desires.

[Nevermark]

> When we went from horses to cars, …

In this situation, we are the horses

[strbean]

This is all speculation based on the development of AGI. If we achieve AGI, yes, it would enable humanity to have a Star Trek-style fully automated space communism utopia. However, that hinges entirely on access to AGI and the fruits of it's productivity being distributed across the population, instead of hoarded by a small group.

If AGI is developed and kept closely guarded, whoever has it will have essentially limitless productivity, and quickly concentrate all wealth and power. They wouldn't even need to engage with markets, they could simply build overwhelming autonomous military power.

[visarga]

Here you are making a mistake - AGI by its nature should know all there is to know, and yet need to make progress by searching for new approaches and discoveries. That doesn't happen all at once, it works field by field, and discoveries actually come from experimental validation. There is no "secretly developing AGI" to "quickly concentrate all wealth and power".

Like bitcoin, you are basically saying someone could outcompute humanity and own the ledger. But in reality the combined research power of humanity, which is necessary for AGI to advance further, is much deeper than any one entity could achieve in isolation. Research is a social process.

[Nevermark]

> the combined research power of humanity, which is necessary for AGI to advance further, is much deeper than any one entity

AGI won't need people to advance further. That is pretty much the functional definition.

But the bar is even lower. Just as today's rich keep increasing the economic distance between themselves and the poor, with the middle and creative classes already feeling that gravity, so will the AGI's - even if they stalled out only as smart as we are, but cheaper in inference mode. (A limitation that is highly unlikely.)

No doubt humans will facilitate AGI activity in our economy and real world for practical reasons for a little while. But at some point, they won't need us physically or socially either.

[visarga]

> AGI won't need people to advance further. That is pretty much the functional definition.

No, AI can ideate as well as any human, but that is not sufficient. Scientists ideate and test. Engineers design and test. There's always a validation stage, where ideas meet the bottleneck of reality.

Are you saying scientists without labs and tools to run experiments on could do science? It's all in the brain or GPU? That is so naive.