[oakpond]

Running dual Pro B60 on Debian stable mostly for AI coding.

I was initially confused what packages were needed (backports kernel + ubuntu kobuk team ppa worksforme). After getting that right I'm now running vllm mostly without issues (though I don't run it 24/7).

At first had major issues with model quality but the vllm xpu guys fixed it fast.

Software capability not as good as nvidia yet (i.e. no fp8 kv cache support last I checked) but with this price difference I don't care. I can basically run a small fp8 local model with almost 100k token context and that's what I wanted.

[lostmsu]

> small fp8 local model with almost 100k token context

Would not fit Qwen3.5 27B would it? That's the SOTA

[oakpond]

This is a fp16 model. That's 54G in weights. I can load it only with fp8 quantization enabled (>= 128k context). I run into this error during generation though: https://github.com/vllm-project/vllm/issues/36350. Looks like an issue with the flash attention backend. But yeah, if you are OK with fp8 quantization on this model, it fits. I expect with 64G VRAM it will fit without quantization