[bob1029]

> Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

In context of genetic programming and other non-traditional ML techniques, I've been having difficulty attempting to locate a simple fitness function that reliably proxies natural language string similarity due to this effect.

For example, say you use something like common prefix length to measure how close a candidate's output string is to an objective string given an input string. The underlying learner will inevitably start doing things like repeating the input verbatim, especially if the input/output training tuples often share a lot of prefixes. So, you might try doing something like reversing the input to force learning to take a less crappy path [0]. The learner may respond degenerately by inventing a string reversing technique and repeating its prior behavior. So, you iterate again and try something like base64 encoding the input. This might take, but eventually you wind up with so many weird hacks that the learner can't make progress and the meaning of the quantities evaporates.

Every metric I've ever looked at gets cheated in some way. The holy grail is probably normalized information distance (approximated by normalized compression distance), but then you have a whole new problem of finding an ideal universal compressor which definitely doesn't exist.

[0]: https://arxiv.org/abs/1409.3215 (Figure 1)

[internet_rand0]

> finding an ideal universal compressor which definitely doesn't exist.

if only we could explain this in "politician" language... too many with too much power think the second coming will deliver the "ideal universal" which doesn't exist

[godelski]

  > I've been having difficulty attempting to locate a simple fitness function that reliably proxies natural language string similarity

Welcome to the curse of dimensionality. The underlying principle there is that as dimensionality increases the ability to distinguish the nearest point from the furthest diminishes. It really becomes difficult even in dimensions we'd consider low by ML standards (e.g. 10-D).

But I think you need to also recognize that you used correct wording that suggests the difficulty. "reliably *proxies* natural language". "Proxy" is the correct word here. It is actually true for any measure. There is no measure that is perfectly aligned with the abstractions we are trying to measure. Even with something as mundane as distance. This naturally leads to Goodhart's Law and is why you must recognize that measures are guides, not answers and not "proof".

And the example you discuss is commonly called "Reward Hacking" or "overfitting". It's the same concept (along with Goodhart's Law) but just used in different domains. Your cost/loss function still represents a "reward". This is part of why it is so important to develop a good test set, but even that is ill-defined. Your test set shouldn't just be disjoint from your training, but there should be a certain distance between data. Even if curse of dimensionality didn't throw a wrench into this situation, there is no definition for what that distance should be. Too small and it might as well be training data. Preferentially you want to maximize it, but that limits the data that can exist in training. The balance is difficult to strike.