Y
Hacker News
new
|
ask
|
show
|
jobs
by
p1esk
221 days ago
5090 has 32GB of RAM. Not sure if that’s enough to fit this model.
2 comments
IceWreck
221 days ago
LlamaCPP supports offloading some experts in a MoE model to CPU. The results are very good and even weaker GPUs can run larger models at reasonable speeds.
n-cpu-moe in
https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...
link
svnt
221 days ago
It should fit enough of the layers to make it reasonably performant.
link
n-cpu-moe in https://github.com/ggml-org/llama.cpp/blob/master/tools/serv...