[mmastrac]

This sort of thing is a lot of fun.

I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.

It's been interesting learning about the balance of factors for performant metal kernels on unified memory.

Should have a repo up on github in the next few weeks.

[saghm]

I'd be interested in seeing this when you put it up! I've managed to get upwards of 300 tokens a second somewhat consistently on the desktop I already had gotten for gaming/personal project stuff several years ago (64 GB RAM, Ryzen 9 7900 X 12-core; I already had a fairly large m.2 SSD and a Radeon 6900 XT to plug into it), but only by severely compromising on the model (a custom quantized Qwen3.5-27B-Q2_K which someone published on huggingface) so that I can use a context size large enough that interfacing through opencode doesn't manage to run out of context while summarizing for compaction and then trigger a second "nested" compaction that instead just seems to cause it to lose all context and ask for a full new set of instructions from scratch. If yours ends up being anywhere as close to the one I've been using in quality, and it works on a 36 GB Mac, it sounds like it would be worth it for me to try out!

(For clarity, I have almost no idea what pretty much any of the terminology for models even means or how it translates to what the actual experience is. My strategy has mostly been using the free tier of OpenCode Zen to ask it stuff about what models and llama.cpp configurations to use in the hopes that I could maybe bootstrap something halfway decent to use locally, and so far what I've managed to get running is mediocre but at least passable)