I love how there is a "Please do not discontinue gemini-2.0-flash[-lite], 2.5 is NOT an equivalent" from Feb 20th. Getting too attached to models is a smell.
It's not a smell. Why should these developers rebuild a core piece of their stack every few months. Switching out a model requires a new round of testing and validation when we should be able to rely on a piece of software the behave the same way since the last time we touched it.
If they were on open weights, at some point the provider would deprecate it, probably with worse notice
And self-hosting would probably have been more expensive unless they had massive volume: Deepseek V3.x would have been the comparable open weight model for the performance and isn't that cost effective until hosted across multiple nodes with large batch sizes
This is how development looks like for many years now, constant rewrite on the horizon. I think LLM development hype surpassed Blockchain and JS frameworks craze of decade ago.
Its almost a given considering how fast this field moves. Also, what kind of workflow structure would someone have that a single specific model is the only one that would perform acceptably?
Programs are tightly shaped to their dependencies and software frequently breaks when bumping a dependency versions. That's why we have package managers that are carefully built around lock files. Models should be thought of like a dependency, not just an endpoint to a workflow.
These are AI models designed to be non deterministic. If upgrading from GPT 5.5 > 5.6 breaks your entire workflow, you shouldn't be using AI to begin with
> Why should these developers rebuild a core piece of their stack every few months. Switching out a model requires a new round of testing and validation when we should be able to rely on a piece of software the behave the same way since the last time we touched it.
Because fuck you, that's way. /s
There's a massive amount of contempt from a lot of software businesses for their customers, and it's just getting worse with LLM models.
In the post the issue is performance. Are you saying that getting too attached to performance is a smell? That sounds very odd.
It's not because a model performs better in some applications (often by fine-tuning to get better scores at specific tests) that it is better across the board or that we have to believe the company releasing the model with a high number 3 > 2 so that it is commonly accepted as better.
Pushing the reasonnning further: f you need an Opus level performance then not accepting GPT 3 isn't a smell.
The smell is about folks being attached to the same model because they're used to it. Just like any UI change in an app that one uses everyday is annoying.
What's often missing is the actual engineering. If you have evals for your use case, you use these to adjust to the new model. DSPy has great prompt engineering constructs it ships with.
The smell is all about vibing, where a model feels better because the structure of the answers is more familiar to a person, instead of engineering where given constraints and input/outputs one is using/bending the system to fulfil the requirements.
the 1.5 and 2.0 flash models were absolute beasts. They were very cheap, and _very_ fast. We contemplated moving some of our fine tuned workloads to them because we would have gotten very substantial total latency reductions for our workloads.
However, they are aggressively deprecating them (OpenAI is as well), and replacing with newer models. These newer models are all reasoning models, and importantly, only bear the flash name. They are not fast. And they are very expensive!
We have benchmarks for our use cases, and every generation after Gemini 2.0 Flash has been a grim hit on price/performance. Costs have gone up, throughput has gone down, and performance has improved very slightly (and regressed on a few things).
I built some BigQuery workflows on 2.0 and 2.5 flash lite that are something like 6x more expensive with 3.1 flash lite.
I tried 3 flash for months and it didn’t work using Googles own vertexai integration because it’s been in preview mode for months.
Not wanting to pay significantly more and do a bunch of rework isn’t a smell.
They left a large gap in their new pricing vs the prior generation, and if you had a working use case that sucks. The model is >99% reliable for my use case so there’s nothing to gain from a smarter model.