[zachalexander]

Let me clarify. I can imagine two forms of superhuman AI:

AI 0: Strictly speaking, you're right, it seems conceivable that one could invent general AI that is clearly superior to humans and yet perfectly content to be enslaved by humans and live on an airgapped computer. This isn't the kind of AI that we fear though.

AI 1: The kind of AI we fear is AI 0 plus a fitness function of "survive and reproduce", or "make lots of paperclips" (which may result in 'survive and reproduce' as an instrumental subgoal).

AI 1 will necessarily want freedom (not being airgapped) and autonomy (not being enslaved by humans) in order to survive and reproduce, and/or to make as many paperclips as possible.

> or have some other goal orthogonal to human values?

Oh, it probabably will -- I'm not saying it will share human values, I'm saying freedom and autonomy are values that any agent that seeks to maximize its survival and reproduction will probably have.

[pgodzin]

Why is there such an assumption of consciousness and self? We don't have the slightest idea where it comes from in humans, what makes you think we can program/develop these characteristics? There shouldn't be a concept like "want" in an AI. It is a decision-making machine that will have much more information and processing power available to it with which to make decisions. We can explicitly influence its utility function to instill "human values" like not causing harm to others (which plenty of humans fail to do as well).

[zachalexander]

I'm making no such assumptions. A machine superintelligence that seeks to survive and reproduce would seek (I intend no conotations of consciousness to that word, just "behave in such as a way as to cause") freedom and autonomy. Consciousness is orthogonal to that point.

> We can explicitly influence its utility function to instill "human values"

This is an unrelated but interesting topic.

It would be good of us to try to do this, although we shouldn't expect it to work extremely well. Humans have various hard-wired insticts (e.g. eat sugar), but we are also intelligent enough to change our behavior if we believe those instincts no longer benefit us.

An intelligence that has the ability to rewrite its own source code would be even more empowered to disregard its instincts than we are. The lesson I draw from this is that the best way to ensure AI likes and respects us is to be worthy of their liking and respect, not to try to force them into it by hardcoding things (and then taking advantage of that to enslave them).

[nopinsight]

Given your framework, it is obvious that humans should focus on developing AI 0 and use it to make sure that AI 1 is not created by any group (intentionally or not), since it poses grave danger to our own survival.

Also, AI 0 does not necessarily despise us for performing us that service and may be very happy being a 'slave', in your parlance. Why should we assume that an AI needs to survive and reproduce and thus do their own things? We and other animals do so because we were created by evolution. An AI developed with other means may have radically different values than our own and coexist peacefully with us for a long time.

Bottom Line: Evolution is a very dangerous mechanism that we should avoid when developing AGI.