[pron]

I don't think so. Who says that the "duplicate" AIs will share the same goals? They'll probably start arguing with one another and form committees. Just like people.

I am poking fun, and I'm not discounting the possibility that AIs could be dangerous (as could people, viruses and earthquakes). But it's very hard not to notice how the singulariters' belief in noncorporal-intelligent deities is a clear, a bit sad, reflection of their own fantasies.

Yudkowsky and friends like to argue for what they call "rationality" (which is the name they give a particular form of autistic thinking imbued with repressed fantasies of power -- or a disaster only they can stop -- which is apparent in all of their "reasonable assumptions"), but their "larger than zero probability" games could be applied to just about any dystopic dream there is. I can say true AI won't happen because a virus epidemic would wipe out the human race first; or over-population would create a humanitarian disaster that will turn us back into savages; or genetic research would create a race of super-intelligent humans that would take over the planet and wipe-out all AI, and I could go on and on.