[zozbot234]

CPU-MoE still helps with mmap. Should not overly hurt token-gen speed on the Mac since the CPU has access to most (though not all) of the unified memory bandwidth, which is the bottleneck.

[abhikul0]

I'll try to use that, but llama-server has mmap on by default and the model still takes up the size of the model in RAM, not sure what's going on.

[zozbot234]

Try running CPU-only inference to troubleshoot that. GPU layers will likely just ignore mmap.