[manwithopinions]

So why are they losing so much money?

Money is made on the subset of inference that is charged at cost + margin via their APIs. API usage is so high because customers are still finding their feet, trying to understand how to measure the value they get from their spend, erring on the side of spend.

Yes, in a world of unmeasured value and tokenmaxxing, inference is profitable on SOTA models because all capacity is being consumed at all times, driving down marginal costs, but what about a world in which capacity isn’t constrained? There are still huge fixed costs.

Even the most optimistic leaks with the current high prices put the margin on API token inference at around 50%. How can SOTA models ever come close to competing on price? Price always matters. Offering the best model with the most brand recognition does not exempt OpenAI from the basic rules of business.

Historically, software has been such a successful business because the margins are incredible, 95%+ in many cases, driven by direct measurable value to customers that dwarfs the cost. A 50% margin at a time when your customers are falling over themselves to spend as much money as they can is not a good sign, it is a very bad sign, it leaves no room to ever achieve traditional technology margins, and inevitably leads to very weak margins.

Inference needs to become an order of magnitude cheaper than the value it delivers to ever have a chance of delivering on this wildly profitable vision. The cheap model providers have a much better chance of achieving that.

Outside of coding, almost every business case for AI doesn’t need above human intelligence, it doesn’t even need human intelligence, or half a human intelligence, a business can extract a lot of value from a machine that has a fraction of a human’s intelligence. Most human work does not use our intelligence, it is rote, a monkey could do it, and that’s where AI will be used most. Who is going to pay $10 per million tokens when they could pay $0.10 to get the same outcomes?

[Tuna-Fish]

> So why are they losing so much money?

Mostly training. Claude didn't just get to be so good at coding by magic, it was suddenly so good because they did truly staggering amounts of RLHF and RLAIF on it. They are still doing that today, on any tasks they can figure out how to evaluate it on. This is capex for them.

Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still profitable). Based on what we know of it's size and architecture, running Opus is not more than 2x more expensive than running Deepseek v4 pro, for which tokens are available at under 10% of the cost of Opus. Again, the reason their margins are 50% is because they are spending so much on things that are not inference, not because inference is expensive.

> The cheap model providers have a much better chance of achieving that.

Anthropic can do it with a push of a button, once they calculate that it will provide them better profit than current pricing.

[vrighter]

you forgot that to not have a knowledge cutoff and fall behing, you need to always be training new models. It matters jack shit if inference is cheap, if you are forced to do training anyway to stay "competitive"

[tliltocatl]

Depends on the application, of course. For "google replacement" they are trying to sell it as - it's absolutely essential, but even then it works like crap. For coding… Maybe it's not so essential? Yea, it would not know about latest libraries, but maybe that's not much of a problem? Then, of course, it is a big question if LLM code generation is worth it at all.

[mikae1]

Hos much does the cutoff matter when models can JFGI?

[overfeed]

The AI companies are attempting displace Google for that sweet ad revenue. Relying on another company's search index means they can squeeze you later for a bigger slice of the pie

[CSSer]

I think people miss this because these companies exist in a space that is new in tech, and that means lots of competition through PR and marketing. When that happens, it’s easy to feel like a company is telling you about everything they’ve been working on or are openly talking about what gives them their edge when in fact the opposite is often true.

[manwithopinions]

> Their margins on inference are >90% today for tokens they sell (plans are hard to count, but still profitable).

That doesn’t make any sense, it doesn’t add up. Have you seen how much money they’re raising and burning? We know that training does not cost tens of billions.

Brockman said OpenAI expects to spend $50 billion on compute this year. OpenAI’s revenue run rate is less than $50 billion for this year! For 90% margins to be possible on inference, you are suggesting that less than $5 billion of that compute spend is inference and over $45 billion of that compute is training.

Anthropic have been desperately trying to juggle capacity by shaping user behavior through peak time usage limits because they are struggling with capacity for inference.

Plan based usage is widely acknowledged to be subsidized, you are probably the only person on earth suggesting that plans are profitable.

[winternewt]

Capacity for inference isn't a cost issue, it's an availability issue. There just isn't enough hardware out there.

[hakfoo]

> Outside of coding, almost every business case for AI doesn’t need above human intelligence, it doesn’t even need human intelligence, or half a human intelligence, a business can extract a lot of value from a machine that has a fraction of a human’s intelligence.

What the business world actually needed isn't intelligence, it's VBA with a bit of polish on it.

Yeah, people want tools to distill reports, and puff nonsense into bigger nonsense, but to a remarkable degree, this doesn't require an LLM. In fact, the alternatives might be preferrable by offering more consistency/repeatability and efficiency (I am so sick of watching a LLM babble for 5 minutes on something a regex would do in 5 seconds).

The genius of the LLM industry is that it avoids "programming anxiety" by hiding it behind a friendly-ish UI and not calling it programming. It's another in the string of innovations like "hide the file system" and "removing user programmability" so we can sell it back to you.

[dada216]

I have deployed very successful LLM based software that reads sales people emails and inserts orders, or stuff akin to orders, in the rest of the systems. Can you write me a regex that parses a messy human email thread and produces a clean JSON with all the order details? It's been working for a year with less than 3% error rate, better than what the humans themselves were doing.

[chairmansteve]

Interesting. A web form completed by the sales people would not have sufficed? I guess it's the backend interfaces to multiple systems that's the difficult part?

[tomnipotent]

A web form assumes the input is or can be made structured, or that you have a user that is willing to fill in a form (vs. writing an email). An LLM is great at taking free-form/unstructured content and mapping it nicely to a structured surface area.

[mxschumacher]

it's astounding how much of human work are tasks like the one you have described here. Dancing around messy interfaces because nobody bothered to properly standardize and automate

[Onavo]

They are overly bloated organizations too. The human costs are tremendously high. Good luck finding ML engineers paid 7-8 figs USD in Asia. Same quality engineers, different market.