[simen]

The serious statement he makes is that humans have complex motivations, and he thinks any superintelligence would as well--in fact it might be a defining feature of intelligence. The Rick and Morty reference was just a humorous example of what might happen, not an actual argument. The subheading was "the argument from complex motivations", not "the argument from Rick and Morty", so let's try and focus on the serious part and not the humorous aside.

[hannasanarion]

But the orthogonality thesis isn't just some idea that may or not be true. It follows directly from Hume's Guillotine, discovered in 1739: no amount of superintelligent reasoning about facts will allow you to derive goals.

No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine.

The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...

[pas]

> it might be a defining feature of intelligence.

But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.