[mcguire]

It's really rather hard to draw any general conclusions from such simple systems:

"In the initial iterations, the wolves were unable to catch the sheep most of the time, leading to heavy time penalties. It then decided that, ‘logically speaking’, if at the start of the game it was close enough to the boulders, an immediate suicide would earn it less point deductions then if it had spent time trying to catch the sheep."

It's as if the scenario you are thinking about involves "assume a machine capable of greater-than-human-level perception, planning, and action" and then set it to optimize a trivially bad function.

How many people do you know with a single goal of "die with as much money as possible", which has a trivial solution: rob a bank and then commit suicide.