Hacker News new | ask | show | jobs
by kzrdude 17 days ago
It might be impossible to run such a large model on the laptop, if I were you I'd first start experimenting with running smaller models, using llama.cpp or lmstudio, just to start understanding how it works and learn about some of the concepts involved - quantization and all!

For example Qwen3.5-4B or Qwen3.5-9B could probably be run cpu-only on some quantizations at more reasonable speeds.

If you want a larger one, you could probably run Qwen3.6-27B or 35B-A3B decently slowly.

Now to be humble - maybe it's impossible to run colibri (this project in the OP) on your computer. Note the performance stories - 0.07 token/s on a 24 GB laptop with 24 CPU threads!