| The smaller model is great at trivial day-to-day tasks. However, when you ask hard things, it struggles; you can ask the same question 10 times, and only get 1 answer that actually answers the question. ...but the larger model is a lot slower. Generally, if you don't want to mess around swapping models, stick with the bigger one. It's better. However, if you are heavily using it, you'll find the speed is a pain in the ass, and when you want a trivial hint like 'how do I do a map statement in kotlin again?', you really don't need it. What I have setup personally is a little thumbs-up / thumbs-down on the suggestions via a custom intellij plugin; if I 'thumbs-down' a result, it generates a new solution for it. If I 'thumbs-down' it twice, it swaps to the larger model to generate a solution for it. This kind of 'use ok model for most things and step up to larger model when you start asking hard stuff' approach scales very nicely for my personal workflow... but, I admit that setting it up was a pain, and I'm forever pissing around with the plugin code to fix tiny bugs, which I would prefer to be spending doing actual work. So... there's not really much tooling out there at the moment to support it, but the best solution really is to use both. If you don't want to and just want 'use the best model for everything', stick with the bigger one. The larger model is more capable of turning 'here is a description of what I want' into 'here is code that does it that actually compiles'. The smaller model is much better at 'I want a code fragment that does X' -> 'rephrased stack overflow answer'. |
I found the performance to be very acceptable for 33b 4 bit on a m3 max with 36gb ram (much faster than reading speed)