[andai]

A year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files).

The model did fine.

Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less.

In this moment, andai was enlightened.

[the-grump]

Sure, that's true until you hit a difficult problem where the smaller models thrash endlessly whereas its big sibling can solve it with one prompt.

[0xedwen]

Claude Code lets you spawn sub-agents mid-session, and I've found framing it adversarially works much better than consensus, and I don't think any of these leaderboards capture it.

[andai]

What do you mean by adversarially here?

[jug]

I agree and this is why I think open models will win in the end. There is just so much to gain on being 10% behind the curve. Especially when the curve is far beyond your needs.

[andai]

Yeah many tasks are already saturated. For small interactive code changes (I call this power coding like power armor), we got there about a year, maybe year and a half ago — with the small models of the time.

And small models have the advantage that they tend to be much faster. This is an interesting factor because, well speed is always a nice-to-have, but if you go from 10 seconds per turn to one second per turn, the activity actually becomes interactive.

It becomes a fundamentally different way of working. You stay active and engaged the whole time. And also because you are "driving", your mental model does not be synchronized from the code base. So you don't need to spend extra time later catching up.

[mejutoco]

is there a benchmark that uses prices or speed as one of the axis, in addition to accuracy?

Best could mean different things to different people.

[artemisart]

Yes same website https://artificialanalysis.ai/models#intelligence-comparison... but they don't have graphs for the individual benchmarks sadly.

[andai]

http://deepswe.datacurve.ai/ has graphs for Cost, for Token Usage, and for Agent Steps. They don't have one for Time, sadly, but Output Tokens and Agent Steps (which appear to produce near-identical rankings) are a decent proxy.

https://cognition.com/blog/frontier-code has a dropdown selector for Tokens, Cost, Time, Agent Steps, and more.

---

As a side note, these two benchmarks appear to be more sensitive at distinguishing supposedly frontier models from each other. But they themselves cannot agree on which is better!

One argues that the other has a flawed methodology (and makes a fair case). However it might also just be that the frontier is a little jagged, and sometimes one model will do better than another.

That's been my experience anyway. When I have a very important task, I always make sure to get a second opinion: I get them both to give it a shot, and then to critique each other's solutions. You can actually apply this at every stage (the review, the planning, the implementation), if you have the patience for it.

(Would be nice if there were a way to automate this process. Maybe with one of the higher level meta-harnesses that runs Claude and Codex as subprocesses. But I haven't looked into that yet...)

At any rate, combinations of models have always been found to do significantly better than a single one (e.g. see Model Alloys, Model Fusion, etc.). So make sure to use them, when appropriate!

Hope this helps.