ollama is another good choice for this purpose. it's essentially a wrapper around llamacpp that adds easy downloading and management of running instances. it's great! also works on linux!
Ollama adding a paid cloud version made me postpone this post for a few weeks at least. I don't object them to make money, but it was hard to recommend a tool for local usage and make the first instruction to go to settings and enable airplane mode.
Luckily llama.cpp has come a long way and was at a point that I could easily recommend as the open source option instead.
Luckily llama.cpp has come a long way and was at a point that I could easily recommend as the open source option instead.