[beefield]

I'm not terribly terrified.

First, I like to think intelligence as the problem space an agent is able to solve. given that this kind of space has approximately infinite dimensions, by any meaningful measure [1] we already have people that are indefinitely more intelligent than an average person, and they seem to not have been turned to paperclip maximizers.

Now, of course, there are different kind of infinities. And yes, there may be an entity coming whose intelligence to humans is like intelligence of humans to ants. And only reason humans destroy ants is because in some rare cases ants happen to annoy humans. And whatever goals this kind of superintelligence may have, it is only out of really bad luck humans may annoy or be of use to such an superintelligence. In the end, maximizing paperclips is really f&cking stupid, so it is quite unlikely such superintelligence would want to maximize paperclips.

[1] I mean meaningful in the sense that you actually can make statements that agent a is N times more intelligent than agent B. Obviously _IQ is not meaningful in this sense, someone with IQ of 120 is not 20% more intelligent than one with IQ 100 in any meaningful sense. It would still in practice take infinite time for infinite amount of average people to finish Project Mannhattan - thus Oppenheimer was infinitely more intelligent than average person.

[comp_throw7]

> maximizing paperclips is really f&cking stupid

Please see the orthogonality thesis for why intelligence is likely to be orthogonal to goals, particularly in artificially-designed agents: http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Ana...

[beefield]

I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals.

Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth.

Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.

[benlivengood]

> I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals.

Lots of humans have pretty despicable goals, including some very intelligent ones.

I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons.

Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.