[nopinsight]

The case is an example of wireheading [1] and illustrates the difficulty of eliciting behaviors we actually desire from complex systems we do not fully understand.

[1] https://wiki.lesswrong.com/wiki/Wireheading

Another lesson: Evolutionary algorithms are really hard to control. Using neural networks developed through evolutionary algorithms means that we are employing a mostly opaque (though not entirely black) box created by a mechanism we can't mentally keep track of in detail. Hope that they are not deployed to control any critical systems until we get a much better grasp of them.

[nopinsight]

Has anyone been able to comprehensively state all of essential human values for a general AI to follow? Thankfully, we do not yet have an operational AGI and it is still quite a bit away from reality. (Narrow AIs we are using do not pose much of a problem because they are limited in capabilities.)