[mrob]

Why does an AI need the ability to "dislike" to calculate that its goals are best accomplished without any living humans around to interfere? Superintelligence doesn't need emotions or consciousness to be dangerous.

[Yoric]

It needs to optimize for something. Like/dislike is an anthropomorphization of the concept.

[mrob]

It's an unhelpful one because it implies the danger is somehow the result of irrational or impulsive thought, and making the AI smarter will avoid it.

[Yoric]

That's not how I read it.

Perhaps because most of the smartest people I know are regularly irrational or impulsive :)

[ben_w]

I think most people don't get that; look at how often even Star Trek script writers write Straw Vulcans*.

* https://tvtropes.org/pmwiki/pmwiki.php/Main/StrawVulcan