[kamranjon]

I think you’re confused - nobody running local models is concerned about SOTA - that’s just marketing hype from large providers - we are interested in data governance, security, control and freedom. You can’t compare hosted services to local inference, these are two very different things, out of principle we aren’t interested in handing our code bases over to untrustworthy third parties.

On your first point, nobody is pasting 64k tokens at once as context, if you are you’ll experience very similar wait times even with hosted providers - context is built up piecemeal and by virtue of being context does not need to be replaced constantly - this is how all agents work.

100% local models are not SOTA, but they are good enough to be incredibly useful if you are a skilled engineer, and I understand the industry is pushing engineers to offload more and more of their work to incentivize higher token spending, but talented engineers can absolutely be just as productive using local models today - it’s just a different style of working that folks who have become accustomed to large providers can’t really comprehend at this point. They’ve vendor-locked themselves into a delusion that they absolutely need SOTA for everything and as a result see everything as black and white.