[MacsHeadroom]

7B parameter models are more than enough for this and run faster than talking pace on even a low end CPU.

Even a finetuned 3B model would be excellent for generative agents and only use about 2GB of RAM to at high speeds on even a single core CPU.

[holografix]

Can you share some examples Of what models you’re referring to?

[T-A]

https://github.com/ggerganov/llama.cpp

https://huggingface.co/mosaicml/mpt-7b