[nitrogen]

The informative first half of the article was fascinating, but it slowly deteriorated from there as I read the argumentative second half. In my view, things took a drastic turn for the worse around this sentence:

Mature genetic engineering, nanotechnology, strong artificial intelligence, and quantum computing, to name but a few, each hold many times the potential for systemic harm to, or destruction of our civilization; and they do so absent the inherent check on their proliferation that was present in the case of nuclear energy...

The suggestion that strong artificial intelligence and quantum computing are more likely destroyers of civilization than nuclear energy seems laughable without further argument. As with nuclear weapons, problems only arise in the application of technology by humans, not in the concepts themselves, and I see far more potential for physical and societal devastation in the application of nuclear weapons than in AI or QC.

[Tuna-Fish]

With nuclear tech, humans have to be the ones pushing the buttons, making pure accidents quite unlikely. If moore's law holds and we make a strong AI, it might theoretically be possible for it to "go FOOM" -- use most of it's thinking power to make itself exponentially smarter, hitting the ultimate physical limits of computing in a relatively short time. If that happens, and the ultimate physical limits of computing are sufficiently far away, the entire human race no longer has any say on it's future -- whatever happens from then on is decided by a singleton entity.

A good overview of the arguments from both sides can be found from the Hanson - Yudkowsky debate on the subject: http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_...

Disclaimer: I have not studied the issue deeply enough to be on either side.

[adrianN]

The Ultimate Physical Limits of Computation:

http://arxiv.org/pdf/quant-ph/9908043

[nitrogen]

Even with a rapidly-developing strong AI, though, humans have to give the initial machine the ability to limitlessly acquire resources and alter its own hardware. Despite the likelihood that a highly intelligent AI could easily convince some humans to do its bidding, the resource acquisition limitation gives me some measure of confidence that strong AI will not be the downfall of our civilization.

I would be more concerned about unexpected emergent behavior in our existing networks as more and more intelligence is added to various systems than a purpose-built self-modifying AI.

[Tuna-Fish]

Assuming it starts at human-level or slightly above, internet access would probably be enough.

Plenty of people make a lot of money over the Internet, and identity theft isn't exactly rare. Anything that's considerably smarter than humans would probably be running on AWS after the first week, without any overt co-operation from it's creators.

I do not hold this to be particularly likely, because I think that the software side of making a mind capable of recursive self-improvement is likely orders of magnitude harder than people seem to think it is. However, if we do succeed in making one, the argument "it needs help from it's creators" is a very weak one -- even a human level one with access to any networking would likely be able to take the ability to improve itself.

[Eliezer]

I don't know what quantum computing is doing on that list, but a mere glance around you will confirm that small increases in intelligence and brainpower, such as that involved between humans and the LCA of humans and chimps, can have rather large effects. Marvin Minsky once said, "Nuclear weapons are not really dangerous because nuclear weapons are not self-replicating." To this I would add that nanotechnology is not really dangerous because nanotechnology is not smarter than you.

[bzbarsky]

Maybe, and maybe not. Worth at least reading http://en.wikipedia.org/wiki/The_Invincible if you haven't for a counter-argument.

[jimrandomh]

The inclusion of quantum computing in that list is stupid, but strong AI is a real and very serious danger. I can't really do the topic justice in a quick comment, so here's a link instead: http://singinst.org/upload/artificial-intelligence-risk.pdf

[olalonde]

Even nuclear energy is arguably a net win for humanity despite its destructive power (at least in an utilitarian way). In fact, it is debatable whether the use of the atomic bomb saved more lives than it took away. [1]

[1] http://en.wikipedia.org/wiki/Debate_over_the_atomic_bombings...