[senko]

Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, I partially agree. Don’t trust the benchmarks, and the Chinese models really are slow and token-inefficient.

However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/

I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.

In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.

[gr_norm]

Yeah, open weights hasn't fully caught up yet, but it's getting very close. And it's certainly passed the point of being reasonably interchangeable with the frontier for (programming) work. Add in the benefits of not being rug-pulled by the frontier labs silently messing with, the knobs on their models or outright denying you the ability to do certain kinds of work (c.f. the HuggingFace fiasco), and they probably come out ahead in several respects.

[michaelbuckbee]

To your slow point: I did a quick eval for a data viz task to do a qualitative comparison and Fable was much faster (by 4x), but I struggle with the "token inefficiency" as a sort of whatever metric.

Kimi was half the cost and produced a near identical output.

https://fr4geiw93g.evvl.io/

On a fairly simple coding task Kimi was 2x the cost and 7x as slow (and gpt-5.6-sol was even cheaper).

https://r26pakjmhv.evvl.io/

[isqueiros]

In my limited anecdotal experience, Kimi K3 is a bit better than Opus 4.8 and Qwen3.8 Max is disastrously bad. It can reason fine, but the moment it tries to do something it gets stuck into long second-guessing loops with no progress. It sometimes refuses to try and debug a problem even after multiple suggestions. I'm sticking to K3.

[ffsm8]

with all the hype around kimi i actually renewed my subscription (last time i tried it out was on 2.5 release) - just the 40€ one.

Its hard to compare that one to my regular sub of anthropic (107€), but everyone always says how much cheaper it is so i thought it was worth a try.

1.5 days later ive went through more then 50% of my weekly usage and decided to renew the regular anthropic sub too.

The API Pricing is definitely cheaper, but anthropics subscription budget seems to equalize that advantage right now. also kimi code feels like claude code from ... september 2025

also - considering how they announced they'd temporarily close subscriptions to make sure they can service customers... i was slightly surprised that there was no issue subscribing, less then 1 day after that announcement. Makes you think if that was just a marketing stunt

[johndoe18637]

Could you check how many tokens the different models spent to do the task? With all the comments about Kimi K3 Not being token efficient I'm curious if your test confirms that

[senko]

For the web app task I mentioned:

* Kimi K3: 9532k input (9172k cached), 114k output - cost $5.5

* Qwen 3.8 Max: 18020k input (17836k cached), 114k output - cost $6.3

* Fable: ~14m input (all cached??), 196k output - cost $30

Correction on my earlier post, Kimi was through Pi, not Kimi Code. For Qwen I used Qwen Code and for Fable I used Claude Code.

Not sure wtf is going on with the Fable stats (a lot tokens, virtuall all of them were cached - I guess heavy system prompt?) but both claude code stats and ccusage tool output match.

[makingstuffs]

Would a fairer test not be to use the same harness for all three? I’d suspect the harness to massively affect token use and optimisation

[JaRail]

Yes and no. It would make the test fair but it would also mean losing any optimizations the vendors have made specifically for their model. They're all trained differently so they should be used differently to get maximum benefit. It makes sense to use each vendor's harness to get the most of their models, and compare the results that way. Additionally, it's not easy to use all the models in all the harnesses. Anthropic's subscription only covers claude code for example.

[makingstuffs]

I don’t know that the methodology of the experiment is testing what it intends to test. With the current method we’re essentially testing each AI lab’s ability to efficiently extract data from their version of a transformer model.

To objectively test all models the harness would need to be the same and ideally independent. Failing that, all three models should be tested in all three harnesses and the output verified on a model x harness level and on an overall aggregated model x all harnesses level.

[maxignol]

> I’d suspect the harness to massively affect token use and optimisation

Yes they do according to databricks -> https://www.databricks.com/blog/benchmarking-coding-agents-d...

That's why I find comparing models on benchmarks only gives the tendency. We should be comparing model x harness to have accurate metrics.

[makingstuffs]

Agreed. An experiment is only as accurate as the methodology is sound.

Testing each lab’s model in its own harness just tells us how well the lab has performed. For the model’s performance we need a control and the only way to get the control is to either test all models in all harnesses or all models in the same harness

[locknitpicker]

> or the web app task I mentioned:

* Kimi K3: (...) cost $6.3

* Fable: (...) cost $30

It's pretty clear that Kimi K3 beats Fable by a long margin.

[vidarh]

I love Kimi for some things, but K3 to me struggles with things Opus 4.8 breezes through. Admittedly stuff that is on the more complex side (code generation bugs in a compiler) to the point that after two days of struggle I stopped Kimi and will have Opus redo its work once I have spare tokens...

This wasn't just being slow - it didn't make forward progress.

For simpler stuff even 2.7 does just fine, though.

[epolanski]

I don't see a difference in capabilities between k3 and fable, but k3 is slow and expensive. Burned through my monthly plan in 3 days.

[Slartie]

I'd guess the slowness is mostly due to there currently being only one provider, Moonshot AI. And they are overwhelmed with demand.

Let's judge the speed of the model when its weights are released and every inference provider on the planet offers it, so demand can spread out a bit.

It's the same topic with token budget comparisons and subscription pricing - don't people understand that this doesn't really matter for open weights models? The pricing is going to be determined by the inference providers, and until they had a chance to evaluate the model on their infra and set token prices accordingly, one doesn't really have anything tangible to compare with other open models nor with closed ones.

[epolanski]

Even if hardware capacity increases, it seems clear it uses way more tokens, so I don't expect parity with other competitors on that front.

On the other hand I expect K3 future refinements to be massive and more efficient.

[Slartie]

The speed at which tokens are crunched, even on the same hardware, differs between models as well. Using more tokens is only a problem if they are processed at the same speed as with a comparison model.

[EwanToo]

Using more tokens is a significant problem if you pay per token?

[Slartie]

Not if the price per token is significantly lower.

Also this arm of the discussion was about speed, not price.

[locknitpicker]

> Using more tokens is a significant problem if you pay per token?

Yoh have posts in this thread suggesting that Fable is 5x more expensive than Kimi K3.

[simjnd]

> the Chinese models really are slow and token-inefficient

This totally depends on the model. Deepseek V4 is very fast and efficient.

[paulddraper]

Not close to frontier.

But yes, it’s cheap.

[simjnd]

The comment I replied to didn't mention anything about being close to frontier, just a blanket statement about Chinese Labs models being slow and inefficient.

People talk about frontier as if it's the only innovation worth pursuing. Deepseek V4 is far from fontier, but it's architecture is super innovative and efficient and what it achieves at that size (especially V4 Flash) is incredible.

[paulddraper]

> Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, the Chinese models really are slow and token-inefficient.

Those are all frontier-competitive models.

[senko]

FWIW my observation was about the models I tested, but I can see how it could be taken as a general statement.

[nxtfari]

Haha, this is awesome. Hadn't heard of it, thank you for the link.

[lumost]

Honestly my anecdotal experience is that fable is benchmaxxed. I have not observed useful gains for Claude since 4.6, with each model iteration making progressively poorer decisions in pursuit of its goal.

The 5.5/5.6 series has performed quite well however. My guess is that my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.

[maxignol]

> my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.

What do you mean by that ? If the model is higher than 50% on swebench pro then it tends to drift from what you like it to do, like DeepSWE benchmarks ?

[lumost]

yes, above 50% I don't observe a significant improvement, the models scoring above 65% like opus4.8 seem worse than gpt5.5/opus4.6. DeepSWE appears to align more closely to my personal experience.

[carterschonwald]

this mirrors my approx experience.