[alwa]

See, my thought would have been the opposite: in a situation like this—where nobody tries the thing because “everybody knows” it’s counterproductive—I’d expect AI literature surveys to confidently assert received wisdom.

It sounds from the quote like even the researchers thought it was a mistake at first… and that on the basis of the literature PLUS their collective professional wisdom. Now, obviously, they did in fact try the thing, so maybe the idea was not quite so wacky as they paint it for the article.

But the point feels similar here as with LLMs and writing: they can do what’s come before pretty well, and they can exhaust a well-specified problem space through sheer muscle; but they seem to be less good at evolving the frontiers of the domain, and I see no mechanism by which to expect that to change.

So I tend to take the opposite lesson: surprises like this renew my hope that there will remain a place in science long into the AI era for meatbags and serendipity and the spirit of curiosity.

[nradov]

LLMs don't rely completely on received wisdom. The training process works in a higher dimensional space so some data points that might seem unrelated to humans end up being clustered close together if there is a hidden or unrecognized relationship.

[alwa]

True, but the clusters do come from somewhere—namely the training set and the input. The more technical the literature, the sparser the prior work; the more answers depend on labwork, the less the bottleneck is purely symbolic reasoning or data-retrieval-at-scale.

To hear it from the researchers, this feels like the sort of finding that, even in retrospect, is non-obvious from existing literature.

I remember hearing scientific progress described in terms of punctuated equilibrium: some Big New Idea, then a bunch of work generalizing that new idea to the rest of the problem space. I could see AI tools speeding up the second type of work: taking a new framework and chewing through everything that came before, in that new light.

But I have a hard time thinking about how the AI techniques could produce novel, surprising outcomes like this one—ones where it’s not just a permutation of existing knowledge, but where it turns out reality actually cuts against the accumulated written knowledge that came before. The “there is magic to be explained here!” aspect of science.

[stevenhuang]

> But I have a hard time thinking about how the AI techniques could produce novel, surprising outcomes like this one

The comparison is not equivalent as a human isolated from the environment and unable to perform experiments would fair the same.

It is through conducting experiments that we make discoveries.

Once AI can hypothesize and run experiments from start to finish I see no reason why novel discoveries won't be made this way.

[ccimmergreen]

Yes, that's only if AI would ask you to validate all prior assumptions to avoid being led by a false premise. I don't see AI or humans bothering to do that.