Hacker News new | ask | show | jobs
by Tuna-Fish 5590 days ago
With nuclear tech, humans have to be the ones pushing the buttons, making pure accidents quite unlikely. If moore's law holds and we make a strong AI, it might theoretically be possible for it to "go FOOM" -- use most of it's thinking power to make itself exponentially smarter, hitting the ultimate physical limits of computing in a relatively short time. If that happens, and the ultimate physical limits of computing are sufficiently far away, the entire human race no longer has any say on it's future -- whatever happens from then on is decided by a singleton entity.

A good overview of the arguments from both sides can be found from the Hanson - Yudkowsky debate on the subject: http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_...

Disclaimer: I have not studied the issue deeply enough to be on either side.
2 comments
adrianN 5590 days ago
The Ultimate Physical Limits of Computation:

http://arxiv.org/pdf/quant-ph/9908043

link
nitrogen 5590 days ago
Even with a rapidly-developing strong AI, though, humans have to give the initial machine the ability to limitlessly acquire resources and alter its own hardware. Despite the likelihood that a highly intelligent AI could easily convince some humans to do its bidding, the resource acquisition limitation gives me some measure of confidence that strong AI will not be the downfall of our civilization.

I would be more concerned about unexpected emergent behavior in our existing networks as more and more intelligence is added to various systems than a purpose-built self-modifying AI.

link
Tuna-Fish 5589 days ago
Assuming it starts at human-level or slightly above, internet access would probably be enough.

Plenty of people make a lot of money over the Internet, and identity theft isn't exactly rare. Anything that's considerably smarter than humans would probably be running on AWS after the first week, without any overt co-operation from it's creators.

I do not hold this to be particularly likely, because I think that the software side of making a mind capable of recursive self-improvement is likely orders of magnitude harder than people seem to think it is. However, if we do succeed in making one, the argument "it needs help from it's creators" is a very weak one -- even a human level one with access to any networking would likely be able to take the ability to improve itself.

link