[zozbot234]

> You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.

On the contrary, that's ultimately an excellent baseline capability for most casual LLM users. Especially if you add the ability to fire off multiple requests over time and work on them concurrently (at least for not-very-long contexts), which is quite natural if one can implement some sort of continuous batching.

Even quantizing the model (with the ensuing loss in quality) is not an absolute requirement, quite unlike e.g. ensuring that a local model can run fully from VRAM on a typical consumer dGPU.

In general, ISTM that people overemphasize real-time or near real-time response which is a rather terrible fit if your goal is efficient LLM inference of near-SOTA models on typical local hardware.