[sudosysgen]

If you read DeepSeek's papers, you'll find a litany of architectural features that allow for a greatly reduced cache hit price by shrinking the size of the KV-cache.

[yfontana]

How come no other big model seems to be able to deliver the same type of extremely low cache cost though, if their techniques are public?

[SXX]

Why dont you think proprietary models use these optimization techniques and way more?

It very much possible Anthropic, OpenAI and Google able to serve their models much cheaper than their current API prices.

They just dont do it because they dont try to ubdercut each other and so far chinese models been percieved behind SOTA.

[jboss10]

I think the "architectural features" are part of the model, not the kv cache. So implementing it would be difficult and expensive.

[natrys]

They can and almost certainly are doing similarly impressive engineering works internally.

They just aren't in any hurry to forward those cost savings to you.

[sudosysgen]

Many of these techniques haven't been published very long ago - it often takes a good 6-8 months for techniques to percolate. But also, they come at a complexity cost and, seemingly, also at a stability cost.

[hnfong]

Also potentially a performance (in terms of output quality) cost. DeepSeek is cheap on a per token basis but lags behind in the benchmarks, perhaps it was a calculated tradeoff.

[petu]

To be fair Deepseek V4 now is just a preview, which likely implies training isn't completed.

[petu]

Deepseek V4 paper is just ~three months old