[benlivengood]

> I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals.

Lots of humans have pretty despicable goals, including some very intelligent ones.

I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons.

Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.