[vanviegen]

That seems unlikely. There are many providers for open models on openrouter. It seems unlikely that they are throwing money away for each token they sell.

Also, there a good technical reasons for inference being much more efficient at scale.

[lowbloodsugar]

Sure. And there’s Lyft and Uber and plenty of others. And Grubhub and DoorDash and uber and how many others. And I don’t even fucking remember how many electric fucking scooter companies, I’m practically falling over scooters! I’m sure they aren’t earning market share by selling at a loss either.

[jolux]

Uber has been profitable since 2023.

[tngranados]

And it was founded in 2009, 14 years without turning a profit

[doodlesdev]

Which, by analogy, would mean you could use subsided tokens for over 10 years and then, and only then, actually purchase your own hardware to do inference, with over a decade of technological advancements to compound.

I'll invest in inference hardware whenever the economics make sense, not because of my prediction that prices will get higher (also, because smaller models keep getting better and they might just suffice for most of my use cases in a few years time).

My biggest worries in terms of cost is in regards to training. Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.

[dijit]

> Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.

Valid takeaway.

"Invest" in the companies that release their models, because the alternative is to reward the people who will capture the market and jack up the prices to unreasonable level.

Unfortunately I think this is a bit like game theory, people will choose the local optimum despite the long term problems with doing so.

[ares623]

It is hard for the engineer to grasp the lengths VCs will go to to grab market and mind share.

[dijit]

The providers on OpenRouter serving open models aren't "throwing money away", agreed.

But that's not the point I'm making. (or, it kind of is, but it's more high level than that).

They're running spot and preemptible GPU instances (60-80% cheaper than on-demand), paying wholesale industrial electricity rates, and running at multi-tenant utilisation densities that make your MacBook look like a bonfire. Of course they're not individually loss-making on inference, they're aggregating cheap commodity compute and skimming a margin, and on paper that's what makes it seem like a good idea, certainly not a loss leader right?

But zoom out a bit; the entire stack is swimming in VC money. OpenRouter itself just raised at a $1.3B valuation backed by a16z. The Chinese models that now account for 36% of all tokens routed through the platform (DeepSeek, Qwen) are priced the way they are because Beijing-adjacent capital has decided market share matters more than margin right now.

So yes, technically no single party is "throwing money away" on each token; they're just all simultaneously subsidising different parts of the stack for strategic reasons. The floor price you're seeing isn't a stable equilibrium, it's a pile of investor money that hasn't entirely finished burning yet.

[vlovich123]

> The floor price you're seeing isn't a stable equilibrium, it's a pile of investor money that hasn't entirely finished burning yet.

All that says is that it gets more expensive in the future as competitors exit the market and sustainability becomes important. That’s why Uber and Lyft were so cheap until they killed taxis. One major difference of course is that some models will remain largely good enough and the incremental cost of running will keep dropping to 0 over time since the hardware needed doesn’t get more expensive and is already purchased.

[dijit]

I think we agree.

I only object to taking current prices as if they are perpetual prices.

[reissbaker]

Most of these folks aren't running on spot/preemptible instances, they're doing 1-2 year reserved rentals. There isn't that much cheap compute floating around and you can be quite profitable on reservations (if you can get them — the compute shortage is real).

I think the question in terms of throwing money away isn't the inference layer: it's whether the companies training open models will be able to financially keep doing so. How long will Moonshot keep releasing future Kimi models? I think there's an interesting wedge they're exploring with being basically a base-model-trainer-as-a-service, i.e. selling rights to Fireworks to sell finetuning services to the Cursors of the world, but it's entirely possible it doesn't pan out.

That being said, Nvidia seems willing to step up to being the base model trainer of last resort via the Nemotron family of open models, since it helps sell more of their hardware — similar to their investments in the CUDA stack to sell hardware (unsurprisingly, Nemotron is designed to run most efficiently on Nvidia hardware, e.g. native NVFP4). So I suspect there will continue to be a pretty good market here.