[oceansweep]

Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.

[Onavo]

I think what OP means is that the "minimum viable product" for a daily use LLM is probably somewhere around e.g. GPT 4o's level of intelligence (YMMV). Below a certain threshold, you are better off using specialized machine learning models rather than general purpose LLMs. It's very difficult to get that level of intelligence fully local on a mobile device without streaming to the cloud.

[mchusma]

I do think this would be interesting if they made these easy to finetune, as I do think this level of intelligence is likely sufficient for many applications and could be extremely cheap to run.