[erikpukinskis]

I'm not too worried about it, because AIs will need to feel and dream and care and suffer and empathize to begin to be as intelligent as a human.

Imagine a human who lacks empathy entirely. That's a disability. They may be able to do some amount of destruction, but at some scale they simply lack the social intelligence necessary to compete with the entire species.

This is the most common mistake I think people make when reasoning about AI: they think human limitations are weaknesses. But they're not weaknesses they're tradeoffs. Natural selection has had a chance to reward all kinds of variations, including more cortex, and less empathy. But we ended up where we are because of tradeoffs.

Any AI which is intelligent in the same way humans are will also have our limitations. Any AI which doesn't have our limitations won't be as smart as us in those respects.

You have to really ask yourself what the difference is between a human with an AI simulator and an AI with a human simulator. In the limit of simulator quality there is none.