[nxtfari]

If you haven’t been really running and testing these models yourself, they are all benchmaxxed. No matter how close they score to frontier on whatever metric, they always fall apart in real world tasks and their token efficiency is ridiculously bad. Fireworks has incredible incentive to make this claim in a headline, because Fireworks hosting K3 for you is pure profit for them, unlike when they host closed source models.

[senko]

Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, I partially agree. Don’t trust the benchmarks, and the Chinese models really are slow and token-inefficient.

However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/

I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.

In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.

[gr_norm]

Yeah, open weights hasn't fully caught up yet, but it's getting very close. And it's certainly passed the point of being reasonably interchangeable with the frontier for (programming) work. Add in the benefits of not being rug-pulled by the frontier labs silently messing with, the knobs on their models or outright denying you the ability to do certain kinds of work (c.f. the HuggingFace fiasco), and they probably come out ahead in several respects.

[michaelbuckbee]

To your slow point: I did a quick eval for a data viz task to do a qualitative comparison and Fable was much faster (by 4x), but I struggle with the "token inefficiency" as a sort of whatever metric.

Kimi was half the cost and produced a near identical output.

https://fr4geiw93g.evvl.io/

On a fairly simple coding task Kimi was 2x the cost and 7x as slow (and gpt-5.6-sol was even cheaper).

https://r26pakjmhv.evvl.io/

[isqueiros]

In my limited anecdotal experience, Kimi K3 is a bit better than Opus 4.8 and Qwen3.8 Max is disastrously bad. It can reason fine, but the moment it tries to do something it gets stuck into long second-guessing loops with no progress. It sometimes refuses to try and debug a problem even after multiple suggestions. I'm sticking to K3.

[ffsm8]

with all the hype around kimi i actually renewed my subscription (last time i tried it out was on 2.5 release) - just the 40€ one.

Its hard to compare that one to my regular sub of anthropic (107€), but everyone always says how much cheaper it is so i thought it was worth a try.

1.5 days later ive went through more then 50% of my weekly usage and decided to renew the regular anthropic sub too.

The API Pricing is definitely cheaper, but anthropics subscription budget seems to equalize that advantage right now. also kimi code feels like claude code from ... september 2025

also - considering how they announced they'd temporarily close subscriptions to make sure they can service customers... i was slightly surprised that there was no issue subscribing, less then 1 day after that announcement. Makes you think if that was just a marketing stunt

[johndoe18637]

Could you check how many tokens the different models spent to do the task? With all the comments about Kimi K3 Not being token efficient I'm curious if your test confirms that

[senko]

For the web app task I mentioned:

* Kimi K3: 9532k input (9172k cached), 114k output - cost $5.5

* Qwen 3.8 Max: 18020k input (17836k cached), 114k output - cost $6.3

* Fable: ~14m input (all cached??), 196k output - cost $30

Correction on my earlier post, Kimi was through Pi, not Kimi Code. For Qwen I used Qwen Code and for Fable I used Claude Code.

Not sure wtf is going on with the Fable stats (a lot tokens, virtuall all of them were cached - I guess heavy system prompt?) but both claude code stats and ccusage tool output match.

[makingstuffs]

Would a fairer test not be to use the same harness for all three? I’d suspect the harness to massively affect token use and optimisation

[JaRail]

Yes and no. It would make the test fair but it would also mean losing any optimizations the vendors have made specifically for their model. They're all trained differently so they should be used differently to get maximum benefit. It makes sense to use each vendor's harness to get the most of their models, and compare the results that way. Additionally, it's not easy to use all the models in all the harnesses. Anthropic's subscription only covers claude code for example.

[makingstuffs]

I don’t know that the methodology of the experiment is testing what it intends to test. With the current method we’re essentially testing each AI lab’s ability to efficiently extract data from their version of a transformer model.

To objectively test all models the harness would need to be the same and ideally independent. Failing that, all three models should be tested in all three harnesses and the output verified on a model x harness level and on an overall aggregated model x all harnesses level.

[maxignol]

> I’d suspect the harness to massively affect token use and optimisation

Yes they do according to databricks -> https://www.databricks.com/blog/benchmarking-coding-agents-d...

That's why I find comparing models on benchmarks only gives the tendency. We should be comparing model x harness to have accurate metrics.

[makingstuffs]

Agreed. An experiment is only as accurate as the methodology is sound.

Testing each lab’s model in its own harness just tells us how well the lab has performed. For the model’s performance we need a control and the only way to get the control is to either test all models in all harnesses or all models in the same harness

[locknitpicker]

> or the web app task I mentioned:

* Kimi K3: (...) cost $6.3

* Fable: (...) cost $30

It's pretty clear that Kimi K3 beats Fable by a long margin.

[vidarh]

I love Kimi for some things, but K3 to me struggles with things Opus 4.8 breezes through. Admittedly stuff that is on the more complex side (code generation bugs in a compiler) to the point that after two days of struggle I stopped Kimi and will have Opus redo its work once I have spare tokens...

This wasn't just being slow - it didn't make forward progress.

For simpler stuff even 2.7 does just fine, though.

[epolanski]

I don't see a difference in capabilities between k3 and fable, but k3 is slow and expensive. Burned through my monthly plan in 3 days.

[Slartie]

I'd guess the slowness is mostly due to there currently being only one provider, Moonshot AI. And they are overwhelmed with demand.

Let's judge the speed of the model when its weights are released and every inference provider on the planet offers it, so demand can spread out a bit.

It's the same topic with token budget comparisons and subscription pricing - don't people understand that this doesn't really matter for open weights models? The pricing is going to be determined by the inference providers, and until they had a chance to evaluate the model on their infra and set token prices accordingly, one doesn't really have anything tangible to compare with other open models nor with closed ones.

[epolanski]

Even if hardware capacity increases, it seems clear it uses way more tokens, so I don't expect parity with other competitors on that front.

On the other hand I expect K3 future refinements to be massive and more efficient.

[Slartie]

The speed at which tokens are crunched, even on the same hardware, differs between models as well. Using more tokens is only a problem if they are processed at the same speed as with a comparison model.

[EwanToo]

Using more tokens is a significant problem if you pay per token?

[simjnd]

> the Chinese models really are slow and token-inefficient

This totally depends on the model. Deepseek V4 is very fast and efficient.

[paulddraper]

Not close to frontier.

But yes, it’s cheap.

[simjnd]

The comment I replied to didn't mention anything about being close to frontier, just a blanket statement about Chinese Labs models being slow and inefficient.

People talk about frontier as if it's the only innovation worth pursuing. Deepseek V4 is far from fontier, but it's architecture is super innovative and efficient and what it achieves at that size (especially V4 Flash) is incredible.

[paulddraper]

> Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, the Chinese models really are slow and token-inefficient.

Those are all frontier-competitive models.

[simjnd]

I guess Deepseek V4 is too now that it's out of preview.

[senko]

FWIW my observation was about the models I tested, but I can see how it could be taken as a general statement.

[nxtfari]

Haha, this is awesome. Hadn't heard of it, thank you for the link.

[lumost]

Honestly my anecdotal experience is that fable is benchmaxxed. I have not observed useful gains for Claude since 4.6, with each model iteration making progressively poorer decisions in pursuit of its goal.

The 5.5/5.6 series has performed quite well however. My guess is that my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.

[maxignol]

> my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.

What do you mean by that ? If the model is higher than 50% on swebench pro then it tends to drift from what you like it to do, like DeepSWE benchmarks ?

[lumost]

yes, above 50% I don't observe a significant improvement, the models scoring above 65% like opus4.8 seem worse than gpt5.5/opus4.6. DeepSWE appears to align more closely to my personal experience.

[carterschonwald]

this mirrors my approx experience.

[jeffreyrogers]

Fable works very well for me on a moderately large codebase. I have had to correct it a few times or point it on the right track, but given how much faster it is at programming than I am that's a very minor issue (and most of these errors are because I underspecified what I wanted in the prompt, I can only think of two cases where it was genuinely wrong... that's a lot better than me in my professional career). Code quality is equal to what I would come up with (and better in areas I'm not familiar with) and the overall software engineering bar is higher because it doesn't get bored when I tell it to do refactors or write integration/regression tests that I would otherwise put off. Also makes it easy to audit code for things like missing audit logging or error notifications that a human would get bored doing.

The product is a fairly standard Ruby on Rails webapp with postgres as the DB. Application complexity is probably a bit higher than average for a webapp. So it's nothing that pushes the boundaries of software engineering, but it is a real product. Token budget has not been an issue for me. I pay for the Max plan ($200/month) and it is well worth it.

[llelouch]

Fable is the clearly best when you have to do real coding.

[epolanski]

Clearly, for you.

I've read the same opinions about Opus and yet it was gpt 5.5 pro via api tackling the hardest problems.

I have used now k3 for 3 days and it has consistently tackled difficult problems sol max could not (orientation optimization algorithms of random 2d shapes on a rectangle for glass cutting).

I have also other beefs with Anthropic models which have been getting smarter and more capable since 4.6, but increasingly worse at acting as assistants, they just want to "do" stuff their own way and ignore instructions often (even simple ones like not to commit, let alone complex ones).

[maxignol]

What is your harness with every model ?

[w4yai]

Define "real coding" ?

For low-level x86 assembly coding, Fable is nowhere near to be as good as Kimi K3.

[gertlabs]

Yes, they are all benchmaxxed, but the question is how benchmaxxed they are relative to each other.

We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.

Data at https://gertlabs.com/rankings

[isityettime]

Although you mention cost, I don't see task cost or total evaluation cost per model in your data. Am I just missing it?

[gertlabs]

The Efficiency tab at https://gertlabs.com/rankings?mode=oneshot_coding (only have cost data for the coding evaluations)

[isityettime]

Thanks!

[chillfox]

I have been testing the various models, and I would not call Fable SOTA.

I can't actually get Fable to do anything. I only work on back-end code, and the moment Fable notices the jwt scope checks on the endpoints it's game over, it refuses to do anything because security is involved.

So for me, Fable is completely useless, the bar is very low, any llm that will actually attempt the task beats it every time.

[RALaBarge]

I've been doing a lot of Trust and Safety and anti-scam stuff recently. When it was first re-released Fable wouldn't do anything for me, now I don't really run into refusals or what it used to do, just stop returning any data at all whatsoever.

Outside topic, but check out the Inkling model -- it is SUPER FAST and does a good job at being a terminal buddy but I would probably offload the real programming or hard tasks to fable or someone else

[user43928]

It has been fine for me for weeks of work in a mobile app, but yesterday I ran into the same limitation.

For implementing a feature where browser credentials need to be handled securely in the app, Fable refuses to work.

[anon373839]

All models are benchmaxxed, period. ”Jagged frontier” is the euphemism du jour, I believe?

Anthropic/OpenAI were touting PhD-level intelligence three years ago. And they’re still shipping models that aren’t smart enough to realize things such as the need to drive the car to the car wash (because they hadn’t yet hill-climbed that particular brain-teaser).

[Wowfunhappy]

> aren’t smart enough to realize things such as the need to drive the car to the car wash

When this first went viral, I immediately tried it on Opus (whatever version was latest at that time), and it got it first try. Tried a few more times in fresh sessions and it got it every time.

Sonnet did screw it up, though.

[nxtfari]

Jagged frontier is not the same as being benchmaxxed. Benchmaxxed is à la Goodhart's Law "when a measure becomes a target, it ceases to be a good measure." Jagged frontier is about how models that seem superhumanly intelligent at one category of tasks (e.g. coding web applications" can seem toddler level or worse at another category (spatial reasoning) because the training corpus doesn't generalize to there.

[senordevnyc]

Anthropic/OpenAI were touting PhD-level intelligence three years ago

No, they weren't. GPT-5 was where OpenAI started talking about PhD-level, and that was less than a year ago.

[JaRail]

(March 2024) Antropic claimed Claude 3 Opus had "graduate-level expert reasoning" with GPQA results of around 60% showing a roughly phd level performance.

(Sept 2024) OpenAI claimed o1 was phd-level in their launch post.

You're kinda both wrong. :)

[rescbr]

They claimed the model was PhD-level, but they never mentioned the university the model graduated from... :)

[pixlmint]

didn't read the article, but just the title felt fishy because of course an inference provider wouldn't mind claiming to provide access to a mythos-class model. So far, GLM 5.2 is still my go-to, even at less than a third the parameter count compared to Kimi K3

[ainch]

It's telling that in their release post, Moonshot themselves said that K3 lags Fable and GPT-5.6 in "user experience". I took that to mean the stuff you can't push directly via RL, what some people call "big model smell".

"Despite being a highly competitive model overall, K3 nonetheless exhibits a noticeable gap in user experience compared with Claude Fable 5 and GPT 5.6 Sol."

https://www.kimi.com/blog/kimi-k3

[typ]

There are some benchmarks I personally trust. But the most reliable indicator other than public benchmarks is the quality of products people are working on, and the model they use for the work. Not a benchmark scoreboard or a one~few shot demo, but the product they have been building for weeks. The reality is, even the diehard advocates who build and sell tooling for open weights, still use proprietary frontier models for their jobs.

[kangarooship]

Sure, the tools and models I use to build real products says a lot about how much I trust them. The closed frontier models are the most familiar, but I'm trying to unlearn my dependence on them. I just don't like them, and I believe AI should be open. In the short term I may lose out on some coding productivity, but in the long term I want to be a better AI engineer anyway. Open model + configurable harness forces me to learn how to get things done more efficiently, and understand how harnesses and models work under the hood.

I'll see what I can ship while not relying on these closed frontier models at all, as unreasonable as that might be. And if I don't get any customers that's on me.

[eru]

I've used Kimi K3 for my (hobby) Linux kernel work recently, because unlike the offerings from OpenAI and Anthropic it didn't flatly declare everything to be a cybersecurity issue.

(And, yes, I know we should blame C. C turns every bug into a cybersecurity nightmare.)

[persedes]

Yeah on internal non coding benchmarks at my company, they are around last years models and don't hold up well.

[selicos]

Clearly the best way to test these is a completely scientific and controlled process of prompting for a creature on a bicycle. Much like testing the fitness of a fish by making it climb a tree, the LLM will show its true colors when presented with the impossible task of rendering an image.

[rw2]

Totally agree with all of this. Most of the problems we solve with AI are not one shot, all the bench marks also miss the human components when testing. Thinking alongside with a human and coming to a solution is what Fable does better than most other models.

[hdgvhicv]

So they aren’t as good as 2024 models? 2025?

[m00dy]

I think this one has greater impact than deepseek, next one for sure, China will take the lead without any question.

[vorticalbox]

I encountered the same issue. Kimi 2.7 looked impressive on paper, but in practice, the code was so riddled with errors that I ended up using GPT-5.5 to fix it.

[jchw]

I've seen this phrase repeated over and over again and I don't think it is true, or at least, to the degree that "benchmaxxing" claims are true, American frontier AI companies are probably about as guilty of it as Chinese AI companies are.

For both GLM 5.2 and Kimi K3 I feel the rough average of the benchmarks gives you a rough idea where they stand. GLM 5.2 was sitting somewhere behind Opus 4.8 but it didn't feel very far away. I've used Kimi K3 via OpenRouter and while I've had limited experience so far, it sure as shit feels like it's right up there with Sol and Fable to me. I am happily able to believe Fable has the edge still, but on a request by request basis it would be pretty easy to get an impression one way or another.

The existing Kimi models were already pretty good so I really just don't find this new model to be that hard to believe. Maybe I'm naive.

[bmitc]

Why is token efficiency a concern with free models?

[dbspin]

They're not free to run, Kimi K3 needs to be run on the cloud, and the quantised versions aren't as capable. Unless you happen to have 3 - 5 TB of VRAM and an 8-node cluster of 8× NVIDIA H100s to run the full fat version. Plus the weights are not yet available to download in any case.

[walrus01]

I agree that quantized versions aren't perfect, but using GLM5.2 as an example, the gap between a BF16 and something like a Q8-K-XL as published by unsloth or a similar Q8 quantization is very minimal. For other "large" LLMs there's a fair number of tests showing that Q8 is about 94% as good at literally half the size in GGUF files on disk, and half the RAM usage. Approx. 1500GB for the BF16 vs 820GB for Q8-K-XL.

[aand16]

"Very minimal" unless the solution to your current task is in that missing %6 of capability.

[bmitc]

Qwen was mentioned in the comment I replied to and can run locally.

[antiloper]

Because you're paying for tokens. Especially output tokens.

[cdud3]

Because that's the only argument left after Kimi beats Fabel in results, price and autonomy.

[hoppp]

Well that sucks that everything is full of misinformation about this subject.

It's better to not even follow the news or benchmarks and just use whatever is available. I make my own judgement.

[Der_Einzige]

You got baited by bad sampling settings.

It's exactly the opposite. Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.

[nl]

> Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.

This seems both arrogantly dismissive ("you are holding it wrong") and incorrect.

Either the OP is using Kimi K3 on Moonshot where is is presumable set correctly (K3 isn't available elsewhere yet), or they are using Kimi K2.x and there has been plenty of time to experiment with this.

[scotty79]

> or they are using Kimi K2.x

That would be terrible because Kimi 2.x is in a different world than Kimi 3

Having opinions on current Kimi model based on 2.x would be like dismissing gpt-5.6-sol because of experiences with gpt4o

[Der_Einzige]

I've earned my right to be arrogantly dismissive since almost the entire field (including the Kimi and qwen team) doesn't know good sampling settings. This is because if you are truly "bitter lesson pilled" you don't think sampling is needed at all.

You can either not believe me and be wrong, or you can (after July 27th) turn on min_p or a better sampler (i.e. top-n-sigma if you got it running via llamacpp) and have it work even better. Up to you.

[dev_hugepages]

What parameter would you advise for min_p?

[selfhoster11]

Check out this’s recent HN submission: https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...

[Der_Einzige]

as temperature approaches infinity, min_p must approach 1 to stay coherent.

Assuming your temp is below 2, min_p of 0.1 is fine (and disable top_p and top_k). You can try 0.05 for more diversity.

Remember that subsequent methods are better, min_p is a mid-tier sampler that just happens to be the best implemented in most inference providers right now.

Also I'm the author of the "conspiracy against high temperature sampling" thing that selfhoster posted in the comments, so you can ask any questions about that piece you want.

[carterschonwald]

i've had trouble finding any anecdotes or data about how to actually set/explore logit sampler settings

[uejfiweun]

All you've said needs the qualifier - "for now!" Look at the trend line. It's clear that if they're not yet at the level of being "good enough" for coding, they will be soon. Sensationalist headlines aside, we all need to be preparing for a world where open models can do pretty much any software tasks you need them to.

[krige]

https://xkcd.com/605/

[Levitz]

Would you bet money that they won't be good enough for coding in, say, one year?

Just as a call to actual consideration, would that seem a smart bet to do? Because I find it really hard to justify dismissal at this point, sure I don't think these things will improve forever and ever, but come on. The goofy Will Smith spaghetti is a little over 3 years old. Three years. Look at where we are at.

[uejfiweun]

This is such a smooth-brain reply, dude. The capability literally exists, today. You're telling me that it's just SO fantastical that people will be able to catch up? Give me a break. Anthropic and OpenAI are not staffed by demigods.

[krige]

That's a whole separate argument you've invented here. And if the capability exists today, why did you talk about some nebulous future?

[uejfiweun]

Is it really so hard to understand that open models are behind and yet they are also continually improving and way better than they were a year ago? Like I'm not sure what your point is here. Are you purposely playing dumb to troll people or something?