[mft_]

Looks impressive, and this size fits achievable home hardware.

That said, if someone would kindly quantise this down for the 64GB paupers, that would be appreciated. (I know there’s likely degradation, but some people reported good results with a 2 bit version of Qwen 3.5 122B, and this is starting from a higher point. Would be interesting to try, at least.)

Edit: someone in the process of doing so: https://huggingface.co/vcruz305/Laguna-S-2.1-GGUF

[verdverm]

The tool I've been using, llm-compressor, can quant models that do not fit in memory (use the sequential pipeline)

https://github.com/vllm-project/llm-compressor

my setup to help you on your way: https://github.com/verdverm/quantr

Though it seems these will not be needed as Poolside has published quants & dflash with their models.

[alfiedotwtf]

Nice! Do you know of any tools that do this for tensorrt models?

[verdverm]

Sorry, I don't, still newer to the quantizing side of things since many are already produced by others.

[yogeshp]

They have also published smaller 33B model called Laguna XS 2.1, its Q4 gguf is 20GB.

https://huggingface.co/poolside/Laguna-XS-2.1-GGUF/tree/main

[mft_]

Thanks for flagging. From the few benchmarks I can find, it looks there or thereabouts with Qwen 3.6-35B-A3B, or maybe a touch below. I'm interested to compare a model that is a big jump larger with pretty impressive benchmarks, but more heavily quantized to fit.

[dofm]

The XS model is genuinely confounding. So far I do not know what to make of it.

Maybe the chat template is broken, maybe not. But I can't get it to think/reason, or at least sometimes it seems to decide not to even when asked. And it is wildly inconsistent when it does it, sometimes thinking out loud.

However, without thinking enabled it is shockingly good at researching using web tool calls. It's also very fast.

I am assuming something is not yet right with the chat template, even using a recent build of llama.cpp, and I guess I will revisit this model again at some point in the future.

[Archit3ch]

> if someone would kindly quantise this down for the 64GB paupers

The Q4_K_M is 75GB. I'm exactly at 64GB, and I wouldn't quantize it further. Instead, do partial weight residency and stream the rest from SSD.

[bitexploder]

I am going to see what I can get out of this streaming from SSD vs. Qwen 27B. Laguna is a seriously impressive model. I already have a lot of success with 27B A3B.

[nunodonato]

unsloth is always here to save the day https://huggingface.co/unsloth/Laguna-S-2.1-GGUF

[andy99]

Seems it’s not fully supported in mainline llama.cpp yet https://github.com/ggml-org/llama.cpp/pull/25165

In the huggingface link they mention building for CPU and for CUDA, does anyone know if that means it wouldn’t be possible to build targeting Vulkan?

[andy99]

Replying to myself, seems this PR was merged into main and it the model does work with a Vulkan backend on my Framework desktop, I’m getting about 220 tok/s prompt processing and 21 tok/s output on the 4-bit quant. This is really a sweet spot imo on this machine between maximizing ram use and still having decent speed due to the expert size. This looks really promising.

[regularfry]

What's your hardware? I've got 64GB RAM and a 4090 here, wondering if it's worth a play.

[andy99]

It’s the Strix halo (AMD) with 128 GB shared memory. The 4bit quant is ~75GB.

Unfortunately I don’t know about the best way of running on an Nvidia gpu, you could try llama.cpp and offloading as many layers as possible into the gpu and using RAM for the rest, not sure if that would slow it down too much though.

[regularfry]

I got usable token rates (10-20tps from memory, so marginal) with the Qwen A10B a while back, well before all the new speculative speedups landed in llama.cpp. There's an unmerged branch which allegedly supports this, but I don't know how well yet. Looks worth investigating but I might give it a few days to see what bugs get shaken loose.

[mft_]

This is interesting: https://github.com/tanishq-dubey/macos-laguna-s2.1

Someone has benchmarked a wide range of different quants.