[ebcode]

Here's the original blog post from last December: https://ebcode.com/blog/2025/12/17/sourceminder/

The 15% number is based on two runs from the ./experiment folder, on an instance from the SWE Bench Pro benchmark. I picked two trajectory logs, from the pro_pilot_openlibrary_haiku_v2 folder, that I consider to be representative of the token savings one can expect. There's huge variance just between individual runs of the same model on the same instance, though, so it's somewhat difficult (and expensive, at API rates) to cut through the noise. But I ran over 80 trials, and I believe that number is accurate.

And I'm publishing the "token science" research in the repo itself, so anyone who's curious about it can review the logs and inspect the python scripts that I used. And if someone out there has the time, inclination, and money, to run some of those experiments themselves, it'd be great to get consensus on some of those results.

And if you'd like to share feedback directly, rather than publicly, my email is on my profile page.