[brucethemoose2]

In addition to what mono said, llama.cpp allocates everything up front with "--mlock"

Llama.cpp (and MLC) have to read the all the model weights from RAM for every token. Batching aside, there's no way around that.

[Art9681]

Mlock is an optional parameter: github.com/ggerganov/llama.cpp/tree/master/examples/main#mlock