[onion2k]

I run Mistral 7b and Llama 3 locally using jani.ai on a 32GB Dell laptop and get about 6 tokens per second with a context window of 8k. It's definitely usable if you're patient. I'm glad I also have a Hugging Face account though.

[Liquix]

seconded - IMHO Jan has the cleanest UI and most straightforward setup out of all LLM frontends available now.

https://jan.ai/

https://github.com/janhq/jan