|
|
|
|
|
|
by skywhopper
806 days ago
|
|
|
SWE-bench Lite is a subset of extremely simple issues from a cherry-picked subset (SWE-bench) of a handful of large (presumably well-run) Python-only projects. Here are some rules they used to trim down the SWE-bench Lite problems: * We remove instances with images, external hyperlinks, references to specific commit shas and references to other pull requests or issues. * We remove instances that have fewer than 40 words in the problem statement. * We remove instances that edit more than 1 file. * We remove instances where the gold patch has more than 3 edit hunks (see patch). See https://www.swebench.com/lite.html |
|
|