[pants2]

Kinda funny that their "cost-vs-performance" chart looks the same as the one for Composer 2.5[1], except that it includes Composer 2.5 at a completely different spot.

What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW.

I'd posit that it's not deliberate deception, but for both companies their training data and benchmarks come from the same dataset (Devin/Cursor interaction logs) so they naturally overfit.

1. https://cursor.com/blog/composer-2-5

[culi]

I think it's also telling that they left out the usual hallmarks of the Pareto distribution: GLM 5.2, Qwen 3.7, Minimax M3, and Mimo 2.5

https://arena.ai/leaderboard/code/webdev/pareto

[petesergeant]

> they left out ... GLM 5.2

They did not.

[oofbey]

Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a single number involves a lot of judgement and trade-offs. I think it's more "you get what you measure" which is true in human organizations too. Define a KPI and people work hard to make it go up, even if it's not quite right or has bad side-effects.

[meowface]

At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideation/etc.) tasks and judge them subjectively, and also see how others judge them subjectively on HN and Twitter.

[girvo]

I use benchmarks…

…that are my own private internal suite on my own code bases where I can judge the output properly

I also measure wall clock time to completion which has been a surprising separator in practice.

[bluelightning2k]

Good observation.

I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.

It is interesting though!

Perhaps in some way this means we should decide which eval set aligns best with our taste?

Back to the blog post. This is an excellent write up of an excellent technical achievement.

I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.

I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.

[swyx]

agent lab, not foundation lab