[simonw]

> The kernel work helped reduce the end-to-end cost of serving the model by 20%, while its experiments increased token-generation efficiency by more than 15%.

If the cost of serving GPT-5.6 just dropped by 20%, does that add up to literally billions of dollars in savings per month?

We know Anthropic spend $1.25 billion renting inference capacity from SpaceX (in two Colossus datacenters) from the SpaceX IPO, but we don't know how much of Anthropic's inference capacity that is (presumably a small fraction, since they were operating on top of AWS and other providers before the SpaceX deal.)

I've not seen any numbers that hint at OpenAI's per-month inference bill, but surely that has to be in the multiple billions of dollars as well.

So 20% is a really, really big deal.

[NitpickLawyer]

~2 years ago gemini2.5 helped write better kernes for itself and (only) reached 1% efficiency gains. Today we're at 20%.

[magicalist]

If you optimize program A and manage to wring out a 1% improvement, and I optimize program B and improve performance by 20%, you can see the problem with trying to infer anything from those two numbers.

Edit: searching for the story now, further bolstering the point is that was 1% in training time [1], and the openAI claim is 20% in end to end inference cost. This is a bad comparison.

[1] https://deepmind.google/blog/alphaevolve-a-gemini-powered-co...

[NitpickLawyer]

> This is a bad comparison.

How so? First, kernel writing (or ML engineering more broadly) is a highly specialised task. Not everyone can do it. It shows that models are getting better and better at (easily verifiable) hard tasks. And you can "hire" that expertise much easier than you can hire the equivalent meatbags. And more importantly you can "fire" them as soon as the task is done. And then hire them 3 months later, when the new model drops. And so on.

Second, 20% gains in inference today gives better end results (i.e. lower overall cost) than 1% in training 2 years ago. Today's models are improving mostly via RL. And RL is highly dependant on fast inference (you want many rollouts for each training scenario). Same for dataset filtering, environment generation, distillation, etc.

[magicalist]

> How so? First, kernel writing (or ML engineering more broadly) is a highly specialised task. Not everyone can do it. It shows that models are getting better and better at (easily verifiable) hard tasks.

No it doesn't. You have no idea how optimised the underlying kernels already were. If a kernel was already optimal, it doesn't matter if you bring a brain the size of a planet, you're not going improve it.

> Second, 20% gains in inference today gives better end results (i.e. lower overall cost) than 1% in training 2 years ago

Your claim (at least in implication) was some kind of trend here. It doesn't matter which one was better to optimise because it "gives better end results" if you're trying to make a trend line out of two unrelated things.

I'm looking for more efficient ways to pack items into packaging. You're looking for more efficient ways to dispatch workers. I report how much faster I can get a shipping container loaded. You report how much less it costs to to get powerlines fixed during a storm.

It's a bad comparison.

[elil17]

> How so?

Without looking at the specific changes made, it is impossible to know whether this represents a real capability improvement. For example, maybe the 20% gain was due to a very obvious/easy to catch inefficiency while the 1% improvement was due to something subtle.

[NitpickLawyer]

I'm not convinced. Both are improvements on something implemented by highly trained/credentialed/successful teams. Something "missed" by either team is impressive either way.

[dust42]

In 2 years from now we will be at 400%. https://xkcd.com/605/ Also, it is called kernels (you have nitpick in your username)

[overgard]

Some numbers: https://www.wheresyoured.at/exclusive-openai-financials/

If those numbers are accurate, I don't think 20% is a really, really big deal. It's like saying "we're digging our grave 20% slower." Ok, but they're still digging!

Or, different analogy, if I'm going broke because I lost my job due to executive AI psychosis, cancelling my netflix subscription doesn't really change the math of not being able to afford rent. It doesn't even really slow it. The amount of money that OpenAI is spending is so absurd that a minor cost saving is like, uh, some progress, but they'd need to do it a lot more to move the needle

[nl]

Those finanicals show OpenAI makes good money on inference.

[overgard]

You might need to reread that mate.

[nl]

> Revenue: $3.7 billion

> Cost of Revenue: $2.65 billion

That's how standard accounting rules for public companies would measure it.

[overgard]

> OpenAI Lost $38.5 Billion In 2025

> 2025 — OpenAI Had $13.07 Billion In Revenue, $34 Billion In Costs and Expenses, and $20.92 Billion In Losses, with a net loss attributable to the company of $38.53 Billion

[dominotw]

imagine writing that on your resume

> reduced inference cost by 20 percent saving company x billion dollars per month

[paxys]

Where are you going to apply to with that resume that’s a step up from your current job though?

[petesergeant]

The other place, but for more money

[bpavuk]

lots of places, actually. not everyone wants to be attached to the Silicon Valley culture, and that line alone will guarantee practically any workplace. that person is going to find out what work-life balance is :)

[paxys]

Sure, but those places don’t need such lofty resumes to begin with.

[speed_spread]

With the right kind of credentials, it's not about need, it's about want. Flip the roles and let yourself become an object of desire, an aspirational hire.

[blovescoffee]

it's kind of hard to save a company billions in operating costs via software without it being part of Silicon Valley (culture)...

[tekacs]

In this case, and I don't mean this critically, I guess it would technically be, "Instructed model to find efficiencies... reducing inference cost by 20% saving company x billion dollars per month."

I have no doubt that further work was required to enable this, but it's still very cool to be possible to say that.

[andai]

I think they meant that GPT-5.6-Sol can write that on its resume.

[da_grift_shift]

Does the model get the credit for its promo packet then? :^)

[hirako2000]

Contributed to. Can't be some IC who made a few nice PRs

[kridsdale1]

Why not. Jeff Dean and John Carmack exist. Both are L10 SWEs.