[abtestdev]

There is no evidence that billions of tokens will ever cost cents. The raw infrastructure cost alone is massive. That's why you see services now charging usage based pricing, because continuing to subsidize inference costs is unsustainable long term.

[bryanlarsen]

cents per billion is only 3 orders of magnitude reduction in cost. I can see 6 or so happening in the next decade or so.

1. Like any commodity, I expect the price to converge to the marginal cost. The marginal cost is the price of electricity which is currently 10% or so of the cost of inference.

2. I expect the cost of electricity to go down an order of magnitude. Solar power and batteries are getting cheaper fast.

3. As the price of electricity starts dominating costs, you can drastically reduce costs by by taking advantage of electricity price variations. Build data centers all around the world and answer requests from the part of the world where the sun is shining.

4. Moore's law

5. Algorithm improvements

6. Moving from general purpose chips to chips that are highly optimized for inference.