[Aurornis]

> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.

This prediction alone isn’t useful at all without a bound on speed and maybe quantization.

You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.

If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.

> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.

You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.

You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.

Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.

[dwa3592]

>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.

What are the LLM standards?

Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.

[Aurornis]

I have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results.

Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing.

It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.

[dwa3592]

Its really easy to argue against local models because when it comes to quality, you can argue using the tokens/sec. and when it comes to speed, you can argue using the parameter count. This is not compared to the frontier stuff but it is the frontier of last year that now runs on a local machine. It was impossible to do this last year.

[andrekandre]

the local models open source harnesses are really improving quite fast; just a few months ago i couldn't get any tool calling to work, and responses were very slow (thinking going on too long etc) but now with some newer models on my macbook air, tool calling works and depending on the model, it returns from thinking fairly swiftly....

[zozbot234]

> You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.

On the contrary, that's ultimately an excellent baseline capability for most casual LLM users. Especially if you add the ability to fire off multiple requests over time and work on them concurrently (at least for not-very-long contexts), which is quite natural if one can implement some sort of continuous batching.

Even quantizing the model (with the ensuing loss in quality) is not an absolute requirement, quite unlike e.g. ensuring that a local model can run fully from VRAM on a typical consumer dGPU.

In general, ISTM that people overemphasize real-time or near real-time response which is a rather terrible fit if your goal is efficient LLM inference of near-SOTA models on typical local hardware.

[imtringued]

>If you’re saying that hardware will catch up by mid-2027, I disagree.

If DDR6 comes out that will actually double the effective memory bandwidth on next generation computers. Most computers will reach a good fraction of a Strix Halo system's memory bandwidth and the next generation of Strix Halo will reach Macbook Pro levels of memory bandwidth. Of course there is no guarantee this will happen by 2027 but DDR6 will probably exist in some form by the end of 2027.