Hacker News new | ask | show | jobs
by saberience 22 days ago
SWE Bench Pro is also gamed and shouldn't be trusted.
2 comments
mnicky 22 days ago
Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
link
Sabinus 22 days ago
Why that benchmark in particular?
link