Hacker News new | ask | show | jobs
by karimf 79 days ago
Depending on the use case, the future is already here.

For example, last week I built a real-time voice AI running locally on iPhone 15.

One use case is for people learning speaking english. The STT is quite good and the small LLM is enough for basic conversation.

https://github.com/fikrikarim/volocal
2 comments
podlp 78 days ago
That’s awesome! I’ve got a similar project for macOS/ iOS using the Apple Intelligence models and on-device STT Transcriber APIs. Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? Maybe we’re not there yet, but I’m interested in a better, local Siri like this with some sort of “agentic lite” capabilities.
link
karimf 78 days ago
> Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets?

I first tried the Qwen 3.5 0.8B Q4_K_S and the model couldn't hold a basic conversation. Although I haven't tried lower quants on 2B.

I'm also interested on the Apple Foundation models, and it's something I plan to try next. AFAIK it's on par with Qwen-3-4B [0]. The biggest upside as you alluded to is that you don't need to download it, which is huge for user onboarding.

[0] https://machinelearning.apple.com/research/apple-foundation-...

link
podlp 77 days ago
Subjectively, AFM isn’t even close to Qwen. It’s one of the weakest models I’ve used. I’m not even sure how many people have Apple Intelligence enabled. But I agree, there must be a huge onboarding win long-term using (and adapting) a model that’s already optimized for your machine. I’ve learned how to navigate most of its shortcomings, but it’s not the most pleasant to work with.
link
Patrick_Devine 78 days ago
Try it with mxfp8 or bf16. It's a decent model for doing tool calling, but I wouldn't recommend using it with 4 bit quantization.
link
Barbing 79 days ago
Brilliant. Hope to see you in the App Store!
link
karimf 79 days ago
Oh thank you! I wasn’t sure if it was worth submitting to the app store since it was just a research preview, but I could do it if people want it.
link
karimf 76 days ago
Ok it's on the app store now: https://apps.apple.com/app/volocal/id6761493288
link