[sampo]

> "I am surprised that so many AI researchers are unable to take the long-term view in this discussion."

In my opinion, the Middle Age alchemists who made gunpowder and first primitive bombs, didn't need to establish research programs to worry about advances in bomb-making leading to the threat of Mutually Assured Nuclear Destruction.

If they had tried (maybe they did?), maybe they would have come up with ideas like very heavy regulation of the trade of saltpeter, so that no one has enough to make a verybigbomb.

I agree that everything you described will happen in the future. But in my opinion we are at a Middle Age alchemist's level in AI research (no offense to AI researchers) and we can safely wait a 100 years, and let those people worry about the existential threat. They will be in a much better position to do so appropriately, because they will know much more than we do. And they will not be late, either.

[rsaarelm]

In this case the worst case would be very fast development of dangerous AI, and I'm not sure the 100 years is anywhere close to the lower bound. Low-end estimates for the computation power needed to run something equivalent to full human cognition are around 1 petaflop [1]. Google's total computing power in 2012 was estimated at 40 petaflops [2]. Of course it's split wide to a wide network of computers, but the human brain we're looking for comparison looks like a pretty parallelizable design. So we seem to already be at point where it might just be the lack of very clever programming that keeps us from getting a weakly superhuman AI running in the Google internals.

It looks like we've got ways to go there though, current programs don't seem to even begin to act anything like an adult human. So if the problem was to engineer an out of the box adult human level AI, we might again assume that there's obviously decades of work left to do before anything interesting can be developed. The problem now is that that's not how humans work. Humans start out as babies and learn their way up to adult cleverness. I can tell that an AI is nowhere near having an adult human intelligence out of the box, but I'm far less sure how to tell that an AI is nowhere near being able to start a learning process that makes it develop from something resembling a useless human baby towards something resembling an adult human in capabilities.

[1] http://www.nickbostrom.com/superintelligence.html [2] https://plus.google.com/+JamesPearn/posts/gTFgij36o6u