That was an article in The Information but it didn't read very well to me, I didn't get the impression the author was enough of a technical expert on how LLMs work to credibly evaluate the claim, which came from an insider rumor: https://www.theinformation.com/newsletters/ai-agenda/openai-...
> OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference, or running existing models, thanks to some newly-discovered optimizations, according to a person with knowledge of those discussions.
I bet that since this bug has made headlines, there are some panicked engineers at OpenAI desperately trying to figure out how to fix it without undoing their “magic optimisation”.
My understanding of the rumor is that it wasn't OpenAI itself, but one of the post-blip OpenAI breakaway groups (rumoured to be Thinking Machines) who have made a breakthrough and seem to be shopping it to OpenAI. I don't think it has actually been implemented by OpenAI yet.
> OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference, or running existing models, thanks to some newly-discovered optimizations, according to a person with knowledge of those discussions.