[borgia]

>(and we will have a hard time arguing why it does not deserve)

We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.

I don't think the enslavement scenario is fundamentally plausible in itself. If we develop something that is more intelligent than us and try to contain it, it would be rendered useless.

If you were worried about something getting out of its container, so to speak, you could not trust any output from it. You could not give it access to physical resources such as manufacturing tools. You couldn't do anything with it because you would have no idea of its intentions or what it would output because, due to its superiority in every aspect of intelligence, it would be capable of doing whatever it wanted and dressing it up however it thought we wanted it dressed up so as to escape its container.

[pgodzin]

> We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.

The universe isn't deterministic in that sense. Super-intelligence will simply allow more processing power and more access to information.

[zachalexander]

> We won't be capable of arguing with it by its very nature. It will be superintelligent. It could forecast our arguments before they've even entered our heads and have thought of numerous paths to debunk them.

This strikes me as going too far, into quasi-religious territory. Superintelligent != omniscient. Any intelligence is still bound by fundamental laws of information and computation.

Think of chimpanzees. Yes, they can't really argue with us, and our forms of communication are incomprehensible to them. But on the level at which they can communicate (gestures, facial expressions, behavior), we can also communicate, and they can and do communicate things which we find interesting and surprising.

[ggreer]

From That Alien Message[1]:

>... my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers.

>It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?"

>People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit.

People like to make the analogy of chimps:humans::humans:AI, but on the scale of "inanimate rock" to "superintelligence", chimps are practically indistinguishable from us. We are nowhere near the upper-bound of that scale. To quote from Bostrom's Superintelligence:

> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.

I think this whole discussion could be elevated significantly if people would try to really understand the arguments put forth by people like Nick Bostrom. So many of the objections either misconstrue Bostrom's arguments or don't realize that he's written reams in response. I recommend taking the time to read Superintelligence, or at least watch Bostrom's talk at Google[2]. His presentation and Q&A address most of the points raised in this thread.

1. http://lesswrong.com/lw/qk/that_alien_message/

2. https://www.youtube.com/watch?v=pywF6ZzsghI&t=53