Hacker News new | ask | show | jobs
by MacsHeadroom 1201 days ago
This app is CPU only and gets good speeds on even mobile phone CPUs. Minimum RAM required is 5GB.
1 comments
sucram1 1201 days ago
Oh wow, any way to do this on Android yet? That would be fun to tinker with, even if it's just the smaller model. Even my older Note 9 has 6GB.
link
MacsHeadroom 1199 days ago
Yes. Starting with the Facebook versions of LLaMA-7B you just quantize the model to 4bit on your desktop (since it takes 14GB of RAM) and then move it to your phone and follow the Android instructions in the repo. https://github.com/ggerganov/llama.cpp/#android

I've seen dozens of screenshots of it running in termux on androids by now at completely usable speeds.

link
sucram1 1199 days ago
Thank you for the link! Insane that this can run on a phone.

As my current potato computer has 8GB of RAM, I'll ask a friend to do it :-)

link