[Centigonal]

If they fixed it, then it wouldn't be SWE-Bench Pro anymore, right? It'd be "SWE-Bench-Pro-Fixed-OpenAI." I think it's better optics for the independence of the benchmark if the OpenAI team lets some third party do the fixing and release the improved benchmark.

...Although OpenAI did exactly that when they released SWE-Bench Verified, so maybe I'm talking out of my butt here.

[kimjune01]

eval shops who depend on these things for a living will have to version their datasets, to prevent contamination