[brookst]

This has nothing to do with queueing.

Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer.

Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is a non sequitur.

[Jensson]

> But the time limit is arbitrary and has nothing to do with other peoples’ workloads

No it has to do with others workloads, now you keep their cache for longer so others will get less. And no its not arbitrary, they run out of memory, if more people do this they will have the dial it down further or run out of capacity.

[brookst]

Explain to me how “every gets 5 minutes, 24/7, regardless of load” is remotely compatible with your statement?

Are you imagining this a fixed MRU where duration scales with usage? Becasue that is not at all what Anthropic documents: https://platform.claude.com/docs/en/build-with-claude/prompt...

You would not get more than 5 minutes if you were the only user in the world. You would not get less at their peak hours.

[searealist]

1) To be clear, it costs you 10x more for uncached input tokens _for the next call_, which are still 5x cheaper than output tokens.

2) Now imagine Anthropic or OpenAI now charge your per minute of reserved VRAM time. It would be more fair if they did. Would you still want to run a tool like this?

[brookst]

Totally agree that if the economics changed, the economics would change.

[toasty228]

> has nothing to do with other peoples’ workloads.

Because ram/memory is free and not in demande at all these days?