[riffraff]

> It probably won't even want to take over. That's a very human trait, and Strong AI wouldn't be human.

There are reason to be worried that prescind from that.

Consider the Paperclip Maximizer[0] example: we build an AI with the sole task of producing paperclips, and it ends up destroying the human race.

[0] http://wiki.lesswrong.com/wiki/Paperclip_maximizer

[lotsofmangos]

I suspect however that the paperclip maximizer as proposed is far more likely to devote time to space travel, as it will soon realise that most of the mass available for paperclip construction exists off-earth, for which it is likely to find human cooperation useful.

This is why it first built Elon Musk.

[efuquen]

Thanks for that reference, definitely going on my reading shortlist. Aside from Musk it got a great endorsement from Russell, who with Norvig coauthored one of the most well known introductory undergraduate texts on AI.

> Nick Bostrom makes a persuasive case that the future impact of AI is perhaps the most important issue the human race has ever faced. Instead of passively drifting, we need to steer a course. Superintelligence charts the submerged rocks of the future with unprecedented detail. It marks the beginning of a new era.