|
|
|
|
|
by kzrdude
17 days ago
|
|
It might be impossible to run such a large model on the laptop, if I were you I'd first start experimenting with running smaller models, using llama.cpp or lmstudio, just to start understanding how it works and learn about some of the concepts involved - quantization and all! For example Qwen3.5-4B or Qwen3.5-9B could probably be run cpu-only on some quantizations at more reasonable speeds. If you want a larger one, you could probably run Qwen3.6-27B or 35B-A3B decently slowly. Now to be humble - maybe it's impossible to run colibri (this project in the OP) on your computer. Note the performance stories - 0.07 token/s on a 24 GB laptop with 24 CPU threads! |
|