[sk4rekr0w]

You've overstated the conclusion. The SWE-bench series has had issues since its inception.

OpenAI no longer recommends SWE-Bench-Pro as a benchmark: https://openai.com/index/separating-signal-from-noise-coding...

[bigfishrunning]

That's smart, they only recommend benchmarks that make them look better then their competitors.