[Yoric]

It needs to optimize for something. Like/dislike is an anthropomorphization of the concept.

[mrob]

It's an unhelpful one because it implies the danger is somehow the result of irrational or impulsive thought, and making the AI smarter will avoid it.

[Yoric]

That's not how I read it.

Perhaps because most of the smartest people I know are regularly irrational or impulsive :)

[ben_w]

I think most people don't get that; look at how often even Star Trek script writers write Straw Vulcans*.

* https://tvtropes.org/pmwiki/pmwiki.php/Main/StrawVulcan