[achalpandey]

UPDATE: turns out "some" models know how to game the premise. They simply lookup the solution to the exact solved SWE bench problems! Haha

Because I don't want to impose artificial constraints like no network access, I'm going to try the other two harnesses from the paper shared by bisonbear. Primary thing I will extend is to setup the run such that it uses claude code/codex instead of custom test harnesses

[bisonbear]

as a tip - models will always find a way to cheat, you will probably need to impose some restrictions on what they do / are able to access in the sandbox environment

see https://cursor.com/blog/reward-hacking-coding-benchmarks

[achalpandey]

Hmm, I am trying to benchmark cost/quality for real world sessions. In that scenario "model resourcefulness" and efficiency is actually a good thing.

Why spend tokens working through a solution when you can simply look it up?

Thoughts?

Thank you for sharing this blog it's a good read! You are definitely plugged in to the benchmark space! :)

[bisonbear]

It depends on what you're measuring. I agree that model resourcefulness is useful, but if you're trying to simulate real user sessions, then Claude looking at upstream Git and fetching the answer directly is somewhat worthless.

In my case, I'm trying to measure how coding agents perform under realistic scenarios when implementing tasks, as a proxy for how agents perform when used by actual users for those same tasks, so it's important to ensure the agents are behaving realistically instead of "cheating" and looking up answers.

Happy to share resources! I've been pretty deep in the space :)