[m12k]

I think a major takeaway here is that balancing a reward system to reward more than a single behavior is really hard - it's easy to tip the scales so one behavior completely dominates all others. It's an interesting lens to use to look at the heuristic reward system humans have built in (hunger, fear, desire, etc). This tends to have an adaptation/numbing effect, where repeated rewards of the same type tend to have diminishing returns, and that makes sense because it protects against "gaming the system" and going for one reward to the exclusion of all others.

[SamBam]

Evolution works in an incredibly complex "fitness landscape," where certain minor tweaks in phenotype or behaviors can affect your fitness in quite complex ways.

Genetic Algorithms attempt to use this same system over extremely simple "fitness landscapes," where the fitness of an agent is defined by programmers using some simple mathematical formula or something.

When the fitness function is being defined in the system by programmers, instead of emerging from a rich and complex ecosystem, then the outcome depends exactly on what the programers choose. If they fail to see the consequences of their scoring algorithm, that's on them. There's nothing really magical going on, they simply failed to foresee the consequences of their choice.

(As someone who has worked with GAs and agent models, this outcome really doesn't surprise me. I would have said "oops, I need to weight the time less" and re-run it, and not thought twice.)

[mcguire]

From the article: (I don't know Chinese, but the animations are clear enough.)

https://www.bilibili.com/video/BV16X4y1V7Yu?p=1&share_medium...

[bserge]

That was my thought, too. They used too few rewards in the first place, but had they used something more complex it would then have become hard to balance it all.

[kzrdude]

leela (lc0) chess also has this problem. People sometimes thinks it wins too slowly (prefers some surefire way to win by 50 moves instead of slightly more risky by 5 moves), or that it plays without tact when in a losing position (it's hard for it to rank moves when all of them lead to a loss, it doesn't have the sense that humans do of still preserving the beauty of the game).

AIs need to learn to feel awkward and avoid it, just like we humans do (even if it feels very irrational at times).

[lancengym]

What do you make of the documentary on AlphaGo where the AI did seemingly suicidal and incomprehensible moves to the human masters but won in the end, baffling everyone? https://youtu.be/WXuK6gekU1Y