[Yiin]

they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.

[BugsJustFindMe]

> they're comparing to similar capability llm models, not humans

25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.

[jchw]

Well first of all, any non-trivial use of LLMs is going to be orders of magnitude more tokens than this, usually multiple millions at minimum. Benchmarks are just benchmarks after all.

Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.

Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...