[RajT88]

Something I do not see represented in these arguments: real world conditions.

Most complex computer systems (which we assume to be the case for a super powerful AI) don't run for very long without requiring manual intervention of some sort. "Aha!" I hear you saying, "The AI will figure out how to reboot nodes and scale clusters and such". OK fine. But then there is the meat space... Replacing hardware, running power stations and all that. Robots... Suck right now compared to humans in navigating the real world and also break down on top of that, just like the systems they would be fixing.

Any Skynet-type scenario would need to be so intelligent it solves all of our engineering problems, so it has no problem designing robots which can reliably fix anything in their system, be it software or hardware.

Insisting that an AGI will be able to figure that stuff out (in ways in which we cannot intervene) is extremely hand-wavey.

[pas]

Just as an example of something that's hard, but not because of lack of technology, real economical surplus, or engineering challenges, yet real things which are still killing millions of people each year: malnutrition, access to clean water, and treatable communicable diseases, like TBC or even HIV or COVID.

Solving those are simple tasks of manufacturing power plants, water treatment plants, fertilizer plants, agricultural machines, logistics of deploying them and keeping them fueled, and persuading people to take the fucking vaccine, use PrEP and use condoms. Money solves these. There's enough people and resources to do this.

The AGI foom argument basically says that the AGI will commandeer the economy of the size of a developed country, well, let's say Germany. (Let's pick Germany, because we already saw what that country was able to do when run by a dictator.)

Insisting that AGI can organize enough people to replace enough broken hardware while manufacturing new hardware doesn't seem that far fetched. (Again, Germany was able to increase its industrial production during WWII while the Allies bombed it constantly for months.)

[adroniser]

You are exploring one possible scenario for how AGI could maintain itself, declaring it extremely unlikely, and then concluding that therefore AGI will be safe. Who says the AGI would alert humans to its actions? Why does a system need robots to execute actions in the real world, when there are plenty of humans it can manipulate to do its bidding?

[RajT88]

You will find I most definitely did not declare a conclusion, upon a careful rereading of my post.