[cantthinkofone]

I think OP means that idealizing ML experiments using contrived data can distort the picture. Real world, ecologically valid results can only be discovered as they emerge when the algorithms are deployed in production. ML algorithms sometimes cook up solutions that can surprise or even disturb their creators.

I'm not sure I exactly agree with this premise. If you read about the principles of chaos engineering, (https://principlesofchaos.org/) it's possible to simulate real world events in testing. And if there's a rigorous mathematical backbone to ML as there clearly is, some determinations about its limitations should be universal for all cases, even if the emergent results in production are unpredictable and could range over intractably many possible outcomes.