[ryandrake]

I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.

[puzzlingcaptcha]

Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.

[trollbridge]

I haven't, since the machine has almost no GPU to speak of. It's an old machine, although in theory it would make sense to pop an R9700 in there and load a bunch of models into RAM and swap them in and out.