[pikwip]

Here's a interesting paper I found that attempts to categorize the mechanisms by which Goodhart's Law operates in the real world. The variants are separated into Causal and Non-causal mechanisms.

https://arxiv.org/abs/1803.04585

[derbOac]

Thanks for posting that. I was going to say -- it's interesting to think about the reasons why Goodhart's Law might hold if it does.

I've always assumed the problem is that the metric is always influenced by other, nontarget variables that become more causally important when the metric becomes a proxy target. So, for example, "gaming the metric" becomes important (in a percent variance sense) after the metric becomes a target. I think the paper's adversarial scenario is closest to this maybe.

They discuss some other factors that seem more relevant to individual cases at any moment in time than an explanation for why a metric's utility might decline over time. In that sense the paper seems to be more about Goodheart-like phenomena in general.

It would be interesting to demonstrate Goodheart's law conclusively with real data in some domains.

[davidmanheim]

Thanks for highlighting our paper!