[godelski]

  > I've been having difficulty attempting to locate a simple fitness function that reliably proxies natural language string similarity

Welcome to the curse of dimensionality. The underlying principle there is that as dimensionality increases the ability to distinguish the nearest point from the furthest diminishes. It really becomes difficult even in dimensions we'd consider low by ML standards (e.g. 10-D).

But I think you need to also recognize that you used correct wording that suggests the difficulty. "reliably *proxies* natural language". "Proxy" is the correct word here. It is actually true for any measure. There is no measure that is perfectly aligned with the abstractions we are trying to measure. Even with something as mundane as distance. This naturally leads to Goodhart's Law and is why you must recognize that measures are guides, not answers and not "proof".

And the example you discuss is commonly called "Reward Hacking" or "overfitting". It's the same concept (along with Goodhart's Law) but just used in different domains. Your cost/loss function still represents a "reward". This is part of why it is so important to develop a good test set, but even that is ill-defined. Your test set shouldn't just be disjoint from your training, but there should be a certain distance between data. Even if curse of dimensionality didn't throw a wrench into this situation, there is no definition for what that distance should be. Too small and it might as well be training data. Preferentially you want to maximize it, but that limits the data that can exist in training. The balance is difficult to strike.