[smcleod]

Weird they're comparing it to really old deepseek v1 models, even v2 has been out a long time now.

[butterfly42069]

My exact thoughts, especially because DeepseekV2 is meant to be a massive improvement.

It seems to be an emerging trend people should look out for that model release sheets often contain comparisons with out of date models and don't inform so much as just try to make the model look "best."

It's an annoying trend. Untrustworthy metrics betray untrustworthy morals.

[bubblyworld]

My barely-informed guess is that they don't have the resources to run it (it's a 200b+ model).

[regularfry]

They could compare to DeepSeek-Coder-V2-Lite-Instruct. That's a 16B model, and it comes out at 24.3 on LiveCodeBench. Given the size delta they're respectably close - they're only just behind at 23.4. The full V2 is way ahead.

[smcleod]

That’s for the larger model, most people running it locally use the -lite model (both of which has lots of benchmarks published)