[_ink_]

> Below: the share of runs that went poking around the web for it. Filthy cheaters!

I can understand that in their benchmark setting they wouldn't want the model to find an existing solution. But in my day to day work, wouldn't I want the model to search online for the best solution? Am I not shooting myself in the foot by preventing it?

[gillesjacobs]

In ML, you want to test general capability of a model (generalizability), because you want it to perform well on unseen tasks. In that benchmark, the literal reference is leaking through web search, the agent can see the matching real codebase online and the commits so that's test set leakage. I know no programmer that was ever paid to rewind an existing codebase to a previous commit and implement a feature/fix a bug that exists in the next commits.