[dylanbyte]

These are high school level only in the sense of assumed background knowledge, they are extremely difficult.

Professional mathematicians would not get this level of performance, unless they have a background in IMO themselves.

This doesn’t mean that the model is better than them in math, just that mathematicians specialize in extending the frontier of math.

The answers are not in the training data.

This is not a model specialized to IMO problems.

[Davidzheng]

Are you sure this is not specialized to IMO? I do see the twitter thread saying it's "general reasoning" but I'd imagine they RL'd on olympiad math questions? If not I really hope someone from OpenAI says that bc it would be pretty astounding.

[stingraycharles]

They also said this is not part of GPT-5, and “will be released later”. It’s very, very likely a model specifically fine-tuned for this benchmark, where afterwards they’ll evaluate what actual real-world problems it’s good at (eg like “use o4-mini-high for coding”).

[UltraSane]

Humans who excel at IMO questions are also "fine tuned" on them in the sense that they practice them for hundreds of hours

[SiempreViernes]

Sure, but nobody is using their IMO score to prove they are superintelligent and pulling it off in wider groups.

[sebmellen]

I’ve seen IMO rank used to justify more than one $100m+ seed round.

[htrp]

Someone was burned by Cognition?

[wanderlust123]

I’m pretty sure that a high score in imo is a sign of high intelligence.

[UltraSane]

And the opposite of ADHD.

[Jensson]

Their hardware isn't fine tuned to it though, it uses the same general intelligence hardware that all other humans use.

So its a big difference if you use a general intelligence system and makes it do well in math, or when you create a specialized system that is only good at math and can't be used to get good in other areas.

[UltraSane]

This IMO LLM isn't using fine tuned hardware either.

[torginus]

From my vague rememberance of doing data science years ago, it's very hard not to leak the training set.

Basically how you do RL is that you make a set of training examples of input-output pairs, and set aside a smaller validation set, which you never train on, to check if your model's doing well.

What you do is you tweak the architecture and the training set until it does well on the validation set. By doing so, you inadvertedly leak info about the training set. Perhaps you choose an architecture which does well on the validation set. Perhaps you train more on examples more like ones being validated.

Even without the explicit intent to cheat, it's very hard to avoid this contamination, if you chose a different validation set, you'd end up with a different model.

[SonOfLilit]

The questions were published a few days ago. The 2025 IMO just ended.

[KoolKat23]

And the model was in lockdown to avoid this.

[YeGoblynQueenne]

>> This is not a model specialized to IMO problems.

How do you know?

[joe_the_user]

Yeah, looking at the GP ... say a sequence of things that are true and plausible. That add your strong, unsupported claim at the end. I remember the approach from when I studied persuasion techniques...

[aprilthird2021]

> The answers are not in the training data.

> This is not a model specialized to IMO problems.

Any proof?

[andrepd]

There's no proof that this is not made up, let alone any shred of transparency or reproducibility.

There are trillions of dollars at stake in hyping up these products; I take everything these companies write with a cartload of salt.

[BoorishBears]

No, and they're lying on the most important claim: that this is not a model specialized to IMO problems.

From the thread:

> just to be clear: the IMO gold LLM is an experimental research model.

The thread tried to muddy the narrative by saying the methodology can generalize, but no one is claiming the actual model is a generalized model.

There'd be a massively different conversation needed if a generalized model that could become the next iteration of ChatGPT had achieved this level of performance.

[AIPedant]

It almost certainly is specialized to IMO problems, look at the way it is answering the questions: https://xcancel.com/alexwei_/status/1946477742855532918

E.g here: https://pbs.twimg.com/media/GwLtrPeWIAUMDYI.png?name=orig

Frankly it looks to me like it's using an AlphaProof style system, going between natural language and Lean/etc. Of course OpenAI will not tell us any of this.

[aluminum96]

OpenAI explicitly stated that it is natural language only, with no tools such as Lean.

https://x.com/alexwei_/status/1946477745627934979?s=46&t=Hov...

[redlock]

Nope

https://x.com/polynoamial/status/1946478249187377206?s=46&t=...

[AIPedant]

If you don't have a Twitter account then x.com links are useless, use a mirror: https://xcancel.com/polynoamial/status/1946478249187377206

Anyway, that doesn't refute my point, it's just PR from a weaselly and dishonest company. I didn't say it was "IMO-specific" but the output strongly suggests specialized tooling and training, and they said this was an experimental LLM that wouldn't be released. I strongly suspect they basically attached their version of AlphaProof to ChatGPT.

[Davidzheng]

We can only go off their word unfortunately and they say no formal math. so I assume it's being eval'd by a verifier model instead of a formal system. There's actually some hints of this b/c geometry in Lean is not that well developed so unless they also built their own system it's hard to do it formally (though their P2 proof is by coordinate bash (computation by algebra instead of geometric construction) so it's hard to tell.

[skdixhxbsb]

> We can only go off their word

We’re talking about Sam Altman’s company here. The same company that started out as a non profit claiming they wanted to better the world.

Suggesting they should be given the benefit of the doubt is dishonest at this point.

[aluminum96]

“they must be lying because I personally dislike them”

This is why HN threads about AI have become exhausting to read

[dandanua]

You don't need specialized tooling like Lean if you have enough training data with statements written in the natural language, I suppose. But the use of AlphaProof/AlphaGeometry type of learning is almost certain. And I'm sure they have spent a lot of compute to produce solutions, $10k is not a problem for them.

The bigger question is - why should everyone be excited by this? If they don't plan to share anything related to this AI model back to humanity.

[fnordpiglet]

I actually think this “cheating” is fine. In fact it’s preferable. I don’t need an AI that can act as a really expensive calculator or solver. We’ve already built really good calculators and solvers that are near optimal. What has been missing is the abductive ability to successfully use those tools in an unconstrained space with agency. I find really no value in avoiding the optimal or near optimal techniques we’ve devised rather than focusing on the harder reasoning tasks of choosing tools, instrumenting them properly, interpreting their results, and iterating. This is the missing piece in automated reasoning after all. A NN that can approximate at great cost those tools is a parlor trick and while interesting not useful or practical. Even if they have some agent system here, it doesn’t make the achievement any less that a machine can zero shot do as well as top humans at incredibly difficult reasoning problems posed in natural language.

[SJC_Hacker]

> I actually think this “cheating” is fine. In fact it’s preferable.

The thing with IMO, is the solutions are already known by someone.

So suppose the model got the solutions beforehand, and fed them into the training model. Would that be an acceptable level of "cheating" in your view?

[fnordpiglet]

Surely you jest. The cheating would be the same cheating as any other situation - someone inside the IMO skipping the questions and answers to people outside then that being used to compete. Fine - but why? If this were discovered then it would be disastrous for everyone involved, and for what? A noteworthy HN link? The downside would be international scandal and careers destroyed. The upside is imperceptible.

Finally, even if you aligned the model with the answers its weight shift of such an enormous model would be inconsequential. You would need to prime the context or boost the weights. All this seems like absurd lengths to go to to cheat on this one thing rather than focusing your energies on actually improving model performance. The payout for OpenAI isn’t a gold medal in the IMO it’s having a model that can get a gold medal at IMO then selling it. But it has to actually be capable of doing what’s on the tin otherwise their customers will easily and rapidly discover this.

Sorry, I like tin foil as much as anyone else, but this doesn’t seem credibly likely given the incentive structure.

[Jensson]

Yet that level of cheating happens all the time because its very unlikely to be discovered. Sometimes its just done by people lower down to increase their own career, since they don't have as much to lose, but cheating does happen and its not that unlikely especially when salaries are this high.

[signatoremo]

Why is "almost certainly"? The link you provided has this to say:

> 5/N Besides the result itself, I am excited about our approach: We reach this capability level not via narrow, task-specific methodology, but by breaking new ground in general-purpose reinforcement learning and test-time compute scaling.

[BoorishBears]

Also from the thread:

> 8/N Btw, we are releasing GPT-5 soon, and we’re excited for you to try it. But just to be clear: the IMO gold LLM is an experimental research model.

And from Sam Altman:

> we are releasing GPT-5 soon but want to set accurate expectations: this is an experimental model that incorporates new research techniques we will use in future models.

The wording you quoted is very tricky: the method used to create the model is generalizable, but the model is not a general-use model.

If I have a post-training method that allows a model excel at a narrow task, it's still a generalizable method if there's a wide range of narrow tasks that it works on.

[torginus]

Since this looks like geometric proof, I wonder if the AI operates only on logical/mathematical statements or it actually somehow 'visualizes' the proof like a human would while solving.