[wren6991]

This seems functionally similar to OpenAI having a step in pricing once you exceed a certain context length (also at 272k aka 2^18 aka 256k).

Having a lot of active context increases the per-token cost (flops issued and bytes read per token out) so it makes sense to pass that cost on to users. I'm actually surprised it's implemented as a hard cutoff instead of a smooth gradient.

[BoorishBears]

Not surprising it's a hard cutoff: they almost certainly have two infrastructure configurations for the two max sequence lengths

Fewer nodes dedicated to prefill per instance, and fewer nodes in total since you don't need to support a higher KV cache.

Disaggregated inference also means they can tune the balance of compute dedicated to prefill seperately from decode

[MrBuddyCasino]

AI infra buildup is so massive that the frontier labs should be able to offer more than one level of context length to incentivize token thriftiness.

One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?

[NoahZuniga]

No, the main issue is that it's hard to communicate pricing where token price increases as token count increases, and they figure they can approximate the parabola well enough with two lines.

[anuramat]

why can't they just increase the price for cached input instead?

[dannyw]

There's the infra cost of having multiple SKUs.

If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.

And my guess is there's probably not a huge amount of customers that want somewhere in between: if you're willing to pay the long context surcharge; you're probably semi-price-insensitive anyway to just use 1M.

[MrBuddyCasino]

> If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.

That seems like it is not an awfully hard problem, because the AI buildout scale is so massive. You simply don't provision too many of the smaller buckets, and when they are fully loaded you route the traffic to the larger buckets. Now you don't have a utilisation problem.

Now of course you can simply organize it that way internally, and present a single price to the customer. Perhaps thats the smarter thing to do.

[pornel]

RAM needed for keeping KV cache around may be the more expensive factor.

[tripplyons]

You only need to cache for tokens that are actually being used. Using 100k tokens of a 256k token window takes the amount of memory as using 100k tokens of a 1m token window.

[pornel]

which is why they charge extra for an option to hold >256K tokens.

The cost may be smoothly variable, but likely there's a bimodal distribution of users who barely use any context and users who push it to the max. Average price across both extremes fits nobody, but averages per kind of workload can be close enough.

Having it as a separate model makes it easier to load-balance the traffic.