[goatlover]

The paperclip maximizer example starts with a human asking an AGI to make some paperclips. That turning into an all-consuming goal at the expense of everything else the AGI would understand humans to care about is the problem with the thought experiment.

However a more complicated example like having the AGI bring about world peace or clean up the environment could have undesirable side-effects because we don't know how to specify what we really want, or have conflicting goals. But that's the same problem we have with existing power structures like governments or corporations.