[gitremote]

> And training is the main money sink, whereas inference is cheap.

False. Training happens once for a time period, but inference happens again and again every time users use the product. Inference is the main money sink.

"according to a report from Google, inference now accounts for nearly 60% of total energy use in their AI workloads. Meta revealed something even more striking: within their AI infrastructure, power is distributed in a 10:20:70 ratio among experimentation, training, and inference respectively, with inference taking the lion’s share."

https://blogs.dal.ca/openthink/the-hidden-cost-of-ai-convers...

[Workaccount2]

They get paid for inference, those tokens might as well be monetary tokens.

[drw85]

This is exactly the problem.

Companies currently are being sold that they can replace employees with little agents that cost $20 to $200 a month.

But then they realize that the $200 last for about 3.5 hours on day 1 of the month and the rest will be charged by the token. Which will then cost as much or more than the employee did, but with a nice guaranteed quota of non determinism and failure rate included.