[swe_dima]

It's scary relying on Google's models.

I have a very price sensitive workload that used to run on flash 2.5 lite - it's deprecated now.

The replacement 3.1 flash lite is a lot more expensive, but now also has a sunset date.

3.5 flash lite is even more expensive.

So the price is rising and you have no choice but to keep paying more and more.

[rayboy1995]

I moved directly from 2.5 flash lite to deepseek v4 flash, its already cheaper and if your prompt caching is good you can save so much more money.

[binary132]

could you explain how to optimize prompt caching or point to a doc about it?

[NeutralForest]

Anything Sam Rose is worth reading: https://ngrok.com/blog/prompt-caching

but the implementation will be up to your provider and harness, for deepseek, they expose some numbers: https://api-docs.deepseek.com/guides/kv_cache/ and Anthropic has a list of actions invalidating your cache: https://platform.claude.com/docs/en/build-with-claude/prompt...

Basically, you avoid anything dynamic: model change, tool change, etc it's also important that your system prompt or main prompt doesn't have non-static data like the date/time/place or someone's name (the person you interact with in a chatbot for example). That should be left to tool call or search.

[samwho]

Sam Rose here. Thank you <3

[NeutralForest]

The man himself, thank you for the articles =)

[samwho]

You are extremely welcome.

[insane_dreamer]

samwho? samrose.

[arjie]

I just put the varying parameters in a trailer prompt and have them change every time. It doesn’t matter because the cache is prefix keyed. You lose caching for the last 20 tokens or so but that’s not a big deal. Moving it to a tool call makes it too slow (needs full roundtrip).

If you’re constructing the prompt you don’t have to jam everything together you can arrange it appropriately.

[NeutralForest]

Yes indeed! Mostly don't put changing data in the beginning or prepend.

[apwheele]

Not an open source, but I discuss it in my book with examples for OpenAI/Anthropic/Gemini, https://crimede-coder.com/blogposts/2026/LLMsForMortals.

All of the models, you need to have a consistent input to get the cache hit. So if you are chatting with a document, and change the system prompt, it will be a cache miss, even if the rest of the items are all the same. If you even pass in the document in not the same order as the prompts, it will be a cache miss. Or if you add tool calls or structured outputs, it will be a cache miss. (Since those generally go at the beginning of the prompt call, not at the end.)

Most of the time when reading documents from URLs directly it will never cache. (Need to typically pass in the bytes directly, or use the provider document store index.)

Gemini has a 4096 minimum token size with the 3 version models before even getting a cache hit. OpenAI it is lower (1024), and is automatic, but only happens in increments of 124. Anthropic can also get cache hits at 1024 tokens, but you need to explicit ask for it (and pay extra).

Caching by default typically lives for 5 minutes since the last cache hit across providers. But some of them you can ask for longer. AWS for Anthropic models can be tricky with multiple endpoint routing, so can get cache misses if it happens to route to a different endpoint.

[ElFitz]

That’s part of why, since Firebase, I’ve tried to never depend on Google products for business, especially not GCP.

Features stay in Beta for ages, whatever that actually means, and released ones get deprecated things fast.

Where some of the competitions treats deprecating entire services as "let’s not put it on your frontpage, put deprecation notices all over the doc, and politely ask new users not to start new project with them".

[pdntspa]

I'm running price-sensitive data extraction workloads on flash 2.5 and its still the king when it comes to accuracy + cost, all the gemini 3 variants perform a bit worse and cost a lot more. Low-key freaking out, ngl

[purvahk08]

you should try out scaledown, it's 29x cheaper and 9% more accurate than gemini https://scaledown.ai/benchmarks/scaledown-vs-gemini

[superkuh]

I felt the same way about openai's text-davinci-002 and code-davinci-002 (gpt-3.5). They were amazing completion models and openai basically dumped them with no equal cost or equal performance replacement. Instead all their models are opaque with no ability to work in completion mode where one actually controls the text input to the model.

These days no company even has completion models where one controls the text input fully. Worthless.

[greatgib]

And somehow, the most annoying is not even the price hike, but it is that is you expect to build a product on any of theirs models, they spend their time being deprecated and you have like to be on the lookup to start from scratch selecting a model and fitting it every year or so... Impossible to have any stability...

[zuzululu]

same I just switched to OpenAI after using flash 2.5 lite for almost everything at our company. We spent thousands just to build this workflow now Google says screw off

[hagen8]

Just switch the model, its not that much effort tbh. And u can also get a cheaper model than 2.5 lite for the same intelligence

[tacoooooooo]

its not always that simple. dropping in a new model is trivial, but highly specific workflows may rely on specific _invisible_ aspects of a model. when that model gets deprecated, the workflow needs to be rebuilt/re-tuned to work with a different model.

google's inability or unwillingness to provide stable timelines for model deprecation makes it risky to build complex workflows using their models

[aitchnyu]

Load-bearing (whoops) quirks were noticeable months back, but haven't most flagship models become predictable and reliable?

[tacoooooooo]

it does seem to be moving in that direction. There were really specific things (large, complex json outputs) that gemini-2.5 flash was basically the only model that seemed capable of reliably for a long period. gpt-5+ has covered the usecase for us now pretty well but still evals slightly below what 2.5 could do

[written-beyond]

100% agreed in the same boat right now. Feeling really screwed over by Google rn

[ActivePattern]

You would be surprised how much of a difference the model makes for certain niche tasks.

For my use case, `gemini-3.1-flash-lite` is ~20% higher accuracy than the next best model of comparable cost (considering both proprietary and open-weight alternatives)

[wasfgwp]

Well it is a bit surprising that 3.1 flash-lite could be better than deepseek-v4-pro (cheaper output and way cheaper cache so might cost less for quite a few use cases).

They are not anywhere close according to pretty much every benchmark (even v4-flash is considerably ahead and its way cheaper than flash-lite). Maybe tuning prompts/tools/etc. might be useful?

[swe_dima]

Gemini flash lite family of models currently has the best ratio for price/speed/intelligence for understanding images, no real alternative AFAIK

[deaux]

"Intelligence" being what, math? Coding? Unfortunately there's a billion use cases for LLMs whose performance is not at all captured by the popular benchmarks they're all trying to maxx.

[whimsicalism]

if you are relying on a model for a business process, it should be simple enough to benchmark on that process

[raducu]

> So the price is rising and you have no choice but to keep paying more and more.

I presume you can't use deepseek?

[bradfa]

There are plenty of 3rd party providers hosting deepseek models, if you don't want to use the 1st party API. 3rd party providers are generally slightly more expensive, but still quite cheap compared to other models of similar vintage and size.

[karolist]

sadly it's not multimodal

[Cyclone_]

They know that there's big enterprises that will have a strong preference to work with another big enterprise instead of relying on a younger company. At least that's why I think they believe they can do this sort of thing and get away with it.

[thinkingtoilet]

All models are increasing in price. Everything up to now has been subsidized by investors, private and public.

[h2aichat]

Opencode Go is just the same. Each month I will I can do less. Dont ask me why?

[viccis]

>So the price is rising and you have no choice but to keep paying more and more.

You can also just write code like you did a year or two ago.

[5701652400]

same here. our production workloads was on Gemini for 2 years. seeing Google unilaterally dropping perfectly fine models and charing you 50x more for worse results is not good.

we are switching to Deepseek.