[andai]

Great article. I'm confused how Sonnet did worse than Haiku though. You mention it did find a bunch of other bugs, just not the ones you were looking for?

9 bugs is probably a bit low of a sample size to get a ranking.

That being said the ranking does end up roughly how you'd expect.

Deepseek is Pro, right? Not Flash? I've been using Flash for a lot of smaller tasks and finding it reasonably good. It's good for "interactive" use. Very fast, does small tasks nearly instantly.

It's also decent for investigating large codebases. I wonder if it could do security work too.

[SwellJoe]

I was surprised by Sonnet's performance, as well. And, it's difficult to say any model is really worse or better based on one attempt across nine bugs (several of which have proven to be intractable for all models, thus far). But, in this particular set of problems, Haiku seems to have done a little bit better. But, self-hosted Qwen 3.6 and Gemma 4 also seem to have done better than Sonnet or Haiku, which is surprising. So, there are surely confounding variables here, but I don't know what they are yet. More testing and more analysis of the data will probably reveal it. It may be that using the Anthropic models in the simpler API harness will unleash their power, maybe there are guardrails baked into the Claude Code system prompt that make the small models too conflicted about right and wrong to answer clearly.

DeepSeek was actually the `deepseek-chat` alias in the API (which dynamically chooses the model based on info I don't know), but when I checked the usage, it was all DeepSeek V4 Pro for the benchmark. I later changed DeepSeek to explicitly use Pro for subsequent experiments, so future runs will be explicitly Pro.

I probably will do a test of smaller models, exclusively, at some point. But, I figured DeepSeek V4 Pro is so cheap, especially given their caching effectiveness and cached input pricing, for my own use I'll probably just use DeepSeek V4 Pro when I need a cheap, fast, near-frontier model.

[andai]

Dang apparently it maps to DeepSeek V4 Flash with reasoning disabled!

https://api-docs.deepseek.com/

[SwellJoe]

No, that's a compatibility thing after they changed the behavior of the aliases.

Or maybe it was calling `reasoner` instead. Whatever it was, the billing definitely showed 100% DeepSeek V4 Pro usage for the benchmark. My only usage was the benchmark, and all usage was Pro. (I only noticed that there was a problem in what the benchmark was calling because in a later run, I started seeing Flash usage, which wasn't what I wanted to test.)

I'm absolutely confident the benchmark results were using DeepSeek V4 Pro. It would be useful to also have Flash data, but the report I linked is all Pro.