[Roxxik]

I have prototyped something similar with ollama some months ago.

Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?

I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.

EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so