[PeterisP]

Of course any powerful intelligence can understand that we wouldn't really want that. The question is why would it care about what we really want? Its core values would be that more paperclips is good, and doing what humans really want is evil if it results in less paperclips.

Currently, we don't know how to properly define "do what I mean / do what we really want" goal in a formal manner; if we had an superpowerful AGI system in front of us ready to be launched today, we wouldn't know how to encode such a goal in it with guarantees that it won't backfire. That's a problem that we still need to solve, and this solution is not likely to appear as a side-effect of simply trying to build a powerful/effective system.

[goatlover]

The paperclip maximizer example starts with a human asking an AGI to make some paperclips. That turning into an all-consuming goal at the expense of everything else the AGI would understand humans to care about is the problem with the thought experiment.

However a more complicated example like having the AGI bring about world peace or clean up the environment could have undesirable side-effects because we don't know how to specify what we really want, or have conflicting goals. But that's the same problem we have with existing power structures like governments or corporations.