I'm super enthusiastic about small models, but let's be realistic. A distillation is not the whole model (and, in fact, a lot of the small distillations on HuggingFace are worse than the base model...most of the Qwen 3.6 Opus/Fable/whatever distillations get weirder on some dimensions than Qwen 3.6 alone, as I understand it).
There are little models that are very good for their size. I say nice things about Gemma 4 damned near every day. But, I'm not writing code with it. I am using it for finding security bugs, though, as the 31b variant is outrageously good at it for its size: https://swelljoe.com/post/gemma-4-exceeds-expectations/ and I'm also using it as a base for my own training experiments, specifically the 12b which is small enough to train a LoRA for on my local hardware so I don't have to rent cloud GPUs. The 12b QAT can run on your Pixel 10 Pro today and is frightfully smart for its size, and has great vision capabilities.
But, I keep saying "for its size". You have to be realistic about what tasks these self-hosted models can do. They are getting better though. Gemma 4 31b is competitive with models 10 times its size from a year ago. That's remarkable, and indicates where things are going.
There are little models that are very good for their size. I say nice things about Gemma 4 damned near every day. But, I'm not writing code with it. I am using it for finding security bugs, though, as the 31b variant is outrageously good at it for its size: https://swelljoe.com/post/gemma-4-exceeds-expectations/ and I'm also using it as a base for my own training experiments, specifically the 12b which is small enough to train a LoRA for on my local hardware so I don't have to rent cloud GPUs. The 12b QAT can run on your Pixel 10 Pro today and is frightfully smart for its size, and has great vision capabilities.
But, I keep saying "for its size". You have to be realistic about what tasks these self-hosted models can do. They are getting better though. Gemma 4 31b is competitive with models 10 times its size from a year ago. That's remarkable, and indicates where things are going.