Hacker News new | ask | show | jobs
by MrBuddyCasino 2 days ago
AI infra buildup is so massive that the frontier labs should be able to offer more than one level of context length to incentivize token thriftiness.

One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?
2 comments
NoahZuniga 2 days ago
No, the main issue is that it's hard to communicate pricing where token price increases as token count increases, and they figure they can approximate the parabola well enough with two lines.
link
anuramat 2 days ago
why can't they just increase the price for cached input instead?
link
dannyw 2 days ago
There's the infra cost of having multiple SKUs.

If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.

And my guess is there's probably not a huge amount of customers that want somewhere in between: if you're willing to pay the long context surcharge; you're probably semi-price-insensitive anyway to just use 1M.

link
MrBuddyCasino 1 day ago
> If you create 3 buckets of inference pods, say, 256k, 512k, and 1M, then you have to worry about filling/dynamically-scaling all of them.

That seems like it is not an awfully hard problem, because the AI buildout scale is so massive. You simply don't provision too many of the smaller buckets, and when they are fully loaded you route the traffic to the larger buckets. Now you don't have a utilisation problem.

Now of course you can simply organize it that way internally, and present a single price to the customer. Perhaps thats the smarter thing to do.

link