[api]

The phenomenon being discussed here is a type of overfitting:

https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart....

The last 50 years or so of managerial practice has been a recipe for overfitting with a brutal emphasis on measuring, optimizing, and stack ranking everything.

I think an argument can be made that this is an age of overfitting everywhere.

[djmips]

Interesting that something similar came up recently where an AI being trained might fake alignment with training goals.