[smeeth]

Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one.

It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.

[acchow]

We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?

[smeeth]

1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model.

2. quantization != native low precision training. a model trained in native ternary should generally outperform a full-precision model quantized after the fact.

even if a ternary model only retains 90-95% of the performance of its fp16 equivalent, who cares? if a 200b ternary model retains most of the capability of the 200b fp16 model while using a fraction of the memory and bandwidth, it can be substantially less efficient per parameter and still dominate a smaller fp16 model under the same hardware budget.

[Aurornis]

> yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance

I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.

Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.

I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.

[benjiro29]

Models are not like a fps counter where losing a few percentage has no actual impact. These percentages may be the difference between a model that writes code and one that goes into loops, and think rm -rf . is a good idea.

There is a reason why most models try to stay in the FP4 or higher range, because the reduced accuracy can have major consequences.

You are better off with a 8b FP4+ model then a 27b Q2 model.

[smeeth]

You're conflating two concepts: native bit width and post-training quantization.

Consider two models: one is 16B and trained natively in 2 bits; one is 8B and trained natively in FP4. These models have the same total number of weight bits, but one has twice as many parameters. There is no real reason the FP4 one should be better just because it's FP4. It might be, but that is an empirical question, not a general rule.

Post-training quantization is another thing entirely. Taking a model trained at higher precision and forcing it down to 2 bits is going to hurt performance, often very badly. But this was never my point.

[busfahrer]

has anyone been able to get bonsai models to run at a proper speed in a x86 windows cpu-only setting, using llama.cpp?

[nkozyra]

No.

But do you need to run every small problem through a 10B-30B model?

We're smashing ants with hammers most of the time. We're asking frontier Opus/Fable models to classify text and build frontend code.

Once we start dissecting these problems into smaller discreet tasks and having the big reasoning models do the tough stuff, we suddenly have an economical system. Not for the company hoping for a big IPO, but for the end user.

[edg5000]

I do about 10 google search queries for every 1 opus/gpt prompt. For google, I don't actually open pages anymore 9 out 10 times; I rely on the AI summary. It's fast and accurate; the trick is that you learn where the boundary is of what you can ask it. Querying information the small model is great at.

Then there might be slow, batch tasks. I can see myself getting 1T of slow RAM one day (in a few years?) and having a slow onsite GLM5.2 doing batch jobs that would be wasteful of my subscription limits, plus sensitive but boring things, such as bookeeping and general admin.

I'd like to to read all my email and al quarterly reporting. But that would have to be a good local model, probably a model simmilar to whatever google search uses, which seems just correct unless you throw serious challenges at it.

[ubercore]

The big hammers buy you more confidence and need less supervision. In pure task execution you _might_ smash the ant with a small surgical hammer, but if you absolutely need it smashed, that's when people still reach for the big hammer. It buys more confidence.

[mdp2021]

> do you need to run every small problem through a 10B-30B model? ... We're asking frontier Opus/Fable models to classify text

Actually probably yes: text analysis (magazine articles) by LLMs in the ~30b .. ~120b range failed miserably (and also randomly - the rare cases of proper interpretation occurred among the failure cases) with the main public models of around one year ago, tried extensively.

So, yes, you can employ an ~80IQ only if you will expect the related quality.

[acchow]

I don’t think you need a 10-30b model for most smartphone use cases.

But I meant to counter gp’s claim that “I can run a 27b model on an iPhone” is kind of pointless and disingenuous. Yes I’m sure someone will come up with a way to run a “27b model” at 0.1 bit quantization on an Apple Watch pretty soon misses the whole point of saying a model is “27b” in capability.

Achieving a parameter count is not the point. And is almost meaningless

[yieldcrv]

Its somewhat good, the prism team’s webgpu demo gave it a couple dozen “kernels” written in Fable 5 and it calls them for almost everything procedural

I feel like these things are experiencing convergent evolution to be like biological brains. The large parameters are merely potentially large parameters and they keep having more and more and smaller active layers, which are themselves quantized down. This is seems analogous to the chemical spiking of neurons and inactive layers of a brain in power and efficiency.

[refulgentis]

There’s a good eval floating around somewhere and tl;dr they’re awesome but the benchmarks are cooked, you’re better off with Qwen 8B Q4 than 27B 1b or ternary.

Thanks for being skeptical, I maintain a llama.cpp-based client and it’s frustrating how high expectations are for local AI bc the median effort level means people mostly assemble their expectations and understanding via marketing soundbites

[flockonus]

There is also the 1bit version @ 3.9 GB that retains 90% of the intelligence - quite a feat!

https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit

[dwa3592]

agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.

[dTal]

>in case they start to ban open source AI development

Is that likely, do you think?

[submain]

NVidia seems to want them. They have a lot of lobbying power.

[dwa3592]

dario has been saying open source models are dangerous. who knows who is listening to him.

[mdp2021]

The one time in which I saw Juergen Schmidhuber in perfect nervous control, "coolness" they may say westward, was when he replied to one member of the audience, "The same observation was made when they invented fire: oh, it's dangerous. But in the end, now it's here (shrugh)".

There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.