[zuminator]

Agreed. Self-aware doesn't necessarily imply having a survival instinct. That instinct came about due to evolution and doesn't require intelligence of any kind, so I don't see that the two are necessarily coincident at all.

To me the real problem wouldn't be the AGI itself, but the madman who asks the AGI how to permanently eliminate of all of the undesirables (as defined by the madman). But I think plain old AIs will (unfortunately) be able to come up with workable answers to that question before they ever develop to the AGI stage. So we might wipe ourselves with plain old AI without ever getting to true self-aware AGI.

[duped]

The madman needs access to the means to eliminate the undesirables. I don't see that as something AI has unbridled access too.

This is the classic hardware v software problem. Software can be arbitrarily smart, but it can only interact with the world using the hardware it's connected to. AI can't do anything more meaningful than what is defined by the devices it controls, and I don't think we're stupid enough as a people to connect AI to a doomsday device.

[amanaplanacanal]

I dunno. Give an AI Internet access and it would have theoretical access to a lot of things not even obviously connected to the internet.