[Hedepig]

I am currently tinkering with this all, you can download a 3b parameter model and run it on your phone. Of course it isn't that great, but I had a 3b param model[1] on my potato computer (a mid ryzen cpu with onboard graphics) that does surprisingly well on benchmarks and my experience has been pretty good with it.

Of course, more interesting things happen when you get to 32b and the 70b param models, which will require high end chips like 3090s.

[1] https://huggingface.co/TheBloke/rocket-3B-GGUF

[jart]

That's a nice model that fits comfortably on Raspberry Pi. It's also only a few days old! I've just finished cherry-picking the StableLM support from the llama.cpp project upstream that you'll need in order to run these weights using llamafile. Enjoy! https://github.com/Mozilla-Ocho/llamafile/commit/865462fc465...

[Hedepig]

Thank you for this :)