[kakugawa]

The prefix cache is a resource shared by all users. This is basically a tragedy of the commons.

[dannyw]

Compute is also a resource shared by all users. When your prompt isn’t cached, your input needs to go through prefill; which is compute-bound.

The cache is discounted for a reason. They WANT you to use it.

[searealist]

They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following:

1) Start charging for VRAM reservations.

2) Charge _other_ customers more.

3) Eat the cost themselves.

[dannyw]

They charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss).

Anthropic (and now OpenAI too for 5.6) prompt caching is not free.

[searealist]

Holding your cache in VRAM for 5 minutes or 3 hours have very different costs to them.

"They already charge me to park my car, why can't I leave it there for a year for the same price as 1 week?"

[brookst]

So why is it 5 minutes at both peak and off peak hours? If they can guarantee 5 minutes at peak hours for everyone, why is it only 5 minutes at off peak?

I don’t think your understanding works.

[fearmerchant]

If true, I suspect Anthropic will begin suspending accounts that use it.

[dannyw]

I highly doubt that.

Keeping your cache warm is a good thing, caching saves compute and electricity.

Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.

[OccamsMirror]

If that was true then Anthropic would have built this functionality in themselves.

[brookst]

They did. On any long running query with large context, tell Claude “give me a status report every 4 minutes to keep the cache warm”. It’s entirely happy to oblige and usually says “good idea, that will keep costs down”