Hacker News new | ask | show | jobs
Show HN: OfflineLLM: Live Voice Chat with DeepSeek, Llama on iOS and VisionOS (offlinellm.bilaal.co.uk)
4 points by bilaal_dc5631 385 days ago
Hi, this is something I've been working on for the past 18 months. There are an abundance of tools to run LLMs locally on desktops (e.g. ollama, LM Studio), but other devices have been left out. This is has been a project to run these models onto iOS and visionOS, which has turned out to work really well. Even an iPhone 14 Pro can quite easily run the 3B parameter version of Llama 3.2. CLIP models also work well too!

It also has a Live Voice Chat which gives a 2-way conversation experience, functionality similar to the cloud-based Gemini Live feature that Google offers.

Under the hood it can run most GGUF models, using a heavily forked and diverged verison of llama.cpp which has helped performance on the mobile devices.

The next steps are to integrate Apple's on device 3B model which hopefully they will be opening up access to at WWDC in a week's time. I'm also in the middle of adding in support for Gemma 3 and Qwen 3.

Let me know what you think!
1 comments
35jelly35 385 days ago
> Even an iPhone 14 Pro can quite easily run the 3B parameter version of Llama 3.2

Wow. I never thought a non-Apple Intelligence phone would be able to run this. Does the phone get hot at all?

Also, how long did it take you to build this and how easy is it to test this in Xcode?

link
bilaal_dc5631 385 days ago
Thanks for the questions.

> Does the phone get hot at all?

It's pretty reasonable and similar to the heat you'll get when playing an intensive game. If you're sensible it's pretty usable.

> how long did it take you to build this

I first started in 2023 and managed to get an MVP out the same year. That was pretty basic and a lot of work has been done since. I don't have an accurate measure of how much time has been spent, but it's had a lot of my attention since I released the first MVP.

> how easy is it to test this in Xcode?

This is pretty nice actually. It runs absolutely fine in the simulator, which is where I do most of my testing. The only time I have to move to a physical device is for performance testing, which isn't a huge drain on productivity.

link