Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
I haven't, since the machine has almost no GPU to speak of. It's an old machine, although in theory it would make sense to pop an R9700 in there and load a bunch of models into RAM and swap them in and out.