[notdefio]

Im less interested in the memory than the memory bandwidth. The current system with 128GB can load pretty big models, but its meaningless unless you want to wait 40 minutes per prompt.

I've had best results with Qwen3.6-35B-A3B, which uses 40GB of memory, but only uses 3 billion parameters per token which helps with throughput.

Until memory bandwidth significantly improves I just can't see myself wanting to use all that memory. Unless it's just to keep a wide variety of models in memory.

[gbuk2013]

I run Qwen3-Coder-Next-UD-Q4_K_XL and other than the initial wait to initialise context (which takes less than 2 minutes) subsequent prompts return in less than a minute, usually less than 30s.

If your performance is significantly slower then you are probably doing it in CPU - there was some fiddling required to get it to use GPU (I use llama.cpp)

[cyanydeez]

I use exactly the setup you describe; it can handle 1-3 agents working if the agent work has IO delays. with MTP models, prefill is quite fast.

Not sure what damage you have, but no one waits 40 minutes per prompt or even a minute; The A3B model loads within 10 seconds and a simple response in opencode is maybe at most a minute, then catches up in 3-5second bursts depending on IO.

I put dynamic context pruning into opencode and tweaked it for 45k-85k context before it shrinks; this lets me get into 500k token sizes and fixed on medium sized github repos.

There's a chance this comment is a skill error:

1. USe llamacpp with a MTP model

2. Use reasoning-budget and reasoning-message

3. Tailor your agent to use the reasoning-message to use subagents and dyanmic compaction.

---

Now you have a reasonable coding agent for cloning, building and extending any github project I've seen so far.

[preya2k]

Are you on Strix Halo? If so, which llama.cpp backend are you using? ROCm on Strix Halo still seems incompatible with MTP models.

[data-ottawa]

I'm able to use ROCm with MTP just fine (Qwen and Gemma).

I use Lemonade and have only the ROCm installed through it (ETA: their llama.cpp version is a fork, not sure what is different vs the main trunk).

[cyanydeez]

yes. I build a docker file from this: https://github.com/kyuz0/amd-strix-halo-toolboxes

It has all the backends you want. Just a script rebuilds it. Then I just put it in a docker compose and whenever we want to update it, we pull the repo and rebuild it.

I just a models.ini like: version = 1 ``` [*] n-gpu-layers = 999 threads = 28 flash-attn = on mlock = off mmap = off fit = off warmup = on jinja = true direct-io = on cache-prompt = true cache-reuse = 256 models-max = 1 models-autoload = 1 timeout = 600 ctx-checkpoints = 256 cache-ram = 0 mlock = on kv-unified = on no-context-shift = on swa-full = on

[Q36-35B-A3B] model = Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf mmproj = mmproj-BF16.gguf rope-scaling = yarn rope-scale = 1.14441 yarn-orig-ctx = 262144 override-kv = qwen35.context_length=int:1000000 ctx-size = 300000 reasoning = auto reasoning-budget = 421 reasoning-budget-message = you're reasoning too much; compress context and use subagents to investigate further spec-type = draft-mtp spec-draft-n-max = 3 batch-size = 16001 ubatch-size = 16001 cache-idle-slots = on temp = 0.6 top-p = 0.95 min-p = 0.00 top-k = 20 repeat-penalty = 1.1 presence-penalty = 0.0 ```

Using subagents, you keep the context size from bloating, and add tooling for the context compaction and it can keep looping. You probably want to clean up opencode's agent; back in the "Prompt Engineering" days there's a lot of unnecessary fluff. The goal of a prompt should be to get the model in the correct starting place, and not, try to give it everything it ever needs to do what you want it to do.

[eigenspace]

For those wondering, the AI Max 395 has around 256GB/s of memory bandwidth, whereas this new 495 has 273GB/s. So a very modest improvement in bandwidth.

[andy99]

Try poolside that came out yesterday https://news.ycombinator.com/item?id=49004937

or Qwen 3.5 122B A10B, both use more memory and still have experts sized for decent speed at the 395’s memory bandwidth at 4bit quantization

[paxys]

If you are waiting 40 minutes for a prompt on a Ryzen max+ you have no idea what you are doing.

[cmrdporcupine]

I have an NVIDIA Spark thingy (the ASUS one) and it's the same problem there, though the prefill side is superior to the Ryzen ones.

But I actually think 128GB is too little. There are some compelling models that are above what can fit in that at reasonable quants (e.g. DeepSeek V4 Flash) but could if the system was 256GB.

If RAM prices weren't so f*cked I think we'd be seeing 256GB and even 512GB unified memory systems becoming quite common. As it is I think it will be 10 years before >128GB becomes feasible on a regular consumer machine for normal people again.

[terhechte]

You can run DeepSeek Flash just fine on 128GB using lower quants. Antirez' DwarfStar supports it really well. The bigger the model, it seems, the less it is affected by high quantization. I know some people are working on quantizing GLM 5.2 so it can fit into 128GB (using additional tricks to select only some experts, etc). There's a lot going on.

[andy99]

> RAM prices weren't so f*cked I think we'd be seeing 256GB and even 512GB unified memory systems becoming quite common

I agree but unfortunately that speaks to the depth of demand right now. Even as more capacity comes online (if it does) I suspect it will all get used up without prices needing to drop. Hopefully on the other side of this 10 years from now it means RAM will be insanely abundant and we’ll all have machines with a couple TB as the same inflation adjusted price as say 16GB a few years ago. Considering single digit MB of RAM was the norm in the early-mid ‘90s its not so far fetched.