Hacker News new | ask | show | jobs
by mnicky 23 days ago
SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.
1 comments
saberience 23 days ago
SWE Bench Pro is also gamed and shouldn't be trusted.
link
mnicky 23 days ago
Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
link
Sabinus 23 days ago
Why that benchmark in particular?
link