|
|
|
|
|
|
by tredre3
64 days ago
|
|
|
You're correct. If you don't have enough RAM for the model, it can still run but most of it will run on the CPU and be continuously reloaded from the SSD (through mmap). A medium MoE like 35B can still achieve usable speeds in that setup, mind you, depending on what you're doing. |
|
|