[throwaway1707]

I had to create an account to respond to this because I am quite convinced these math problems they are "solving" are pure marketing. Why is it only GPT doing this, why not Claude? Why does Terrance Tao do marketing for OpenAI? I suspect OpenAI has hired math researchers to solve obscure problems and put them in their training set, purely for marketing reasons.

There was a good comment on the Pelican bicycle svg yesterday about how these models aren't getting much better beyond what the companies focus training them on. I think that's what's happening in this case too, they probably put this in the training set.

[raincole]

It's such a weird train of thoughts lol. You're using the fact that

- Claude isn't doing that

as evidence to support the assumption that

- it's a marketing trick

Which is obviously non sequitur, as if it were a marketing trick, Anthropic could do it too. Anthropic isn't known for not spending on marketing.

Honestly, nowadays I question human's reasoning ability more than I question AI's.

[phillip_kerger]

I (author of the paper) can't speak for others, but I have no affiliation whatsoever with OpenAI and have not received anything from them (I even pay for my subscription lol). I've thrown this problem at each model with new model releases, and 5.6 was the first one to solve it, so that's my side of that story.

[throwaway1707]

Thanks for responding. I don't personally believe you have affiliation, more that OpenAI saw the problem was being worked on, gathered training data for solving it and training the model on it, then eventually you found it through the model (and now it's a big impressive thing for OpenAI)

Part of me agrees with the other comments that it sounds absurd, especially because of how involved/intricate your prompting was. But given you have been prompting this problem for a year, OpenAI could easily have seen your attempts/progress, and worked towards solving it with human intelligence that the agent was then trained on. Given the lengths these companies go to for training these things (Meta literally reorganizing their engineering department to provide training data for engineering problems), and also how much difficulty these models have with problems outside their dataset, I really have to wonder if it's the model being "intelligent" or if it has been trained on it

[threethirtytwo]

Terrence Tao getting paid by openAI is, to you, the most probable conclusion... much more so then the LLM actually being able to come up with math proofs?

[throwaway1707]

Terrance Tao has for a fact appeared in promotional material for OpenAI. Based on my Googling the consensus seems to be he is paid for it, but I cannot confirm that.

I do think it's very likely that OpenAI pays for solutions like these to put in the training set, and then we get material like this Reddit thread. They market themselves as selling "intelligence", and solving these math problems is something people view as highly intelligent. I'm not a mathematician, so I cannot fully judge it, but based on my experience using LLMs for novel problems in other domains, they seem to really struggle with things that aren't common. That leads me to believe they train for specific outcomes like this. Also, there are a lot of jobs out there for data annotation, including software problems (Meta has basically reorganized its entire engineering department to create training data for coding problems).

This comment on the Pelican svg better articulates what I'm getting at: https://news.ycombinator.com/item?id=48950883

[Jweb_Guru]

You can go through my commenter history and know I'm no fan of LLMs. I don't overstate LLM capabilities and am highly skeptical of them in general. 5.6 Pro is genuinely pretty good at certain kinds of math problems that just require trying out lots and lots of solutions, mostly because it's stubborn and can run a bunch of instance in parallel. It is NOT good at coming up with unique ideas or recognizing when its proof approach is doomed, and if the correct approach isn't in its "bag of tricks" for tackling a specific kind of problem, it is not going to get it without a lot of guidance. That said: I 100% believe that it's solved the problems people are claiming that it solved.

The way you should read this is (IMO) not that LLMs have somehow achieved AGI, but that a lot of mathematical research is more about knowing a huge amount of mathematical background, being stubborn, and getting lucky with an approach than it is about brilliant insight. Many people who don't think of themselves as particularly mathematically gifted could have made progress on these problems if they were given enough time and were interested enough. What's notably different about 5.6 (and born out in benchmark after benchmark) is that it does seem to genuinely "reason" through stuff at all -- without that, persistence is pretty worthless because the LLM just goes wildly off the rails if it's put to work for long enough (5.6 itself will still do this if it can't find an answer in a reasonable amount of time).

[Jweb_Guru]

> Why is it only GPT doing this, why not Claude?

Because Claude can't do it. Anyone who tells you that Fable is better than GPT 5.6 at pure math is lying to you.

[frozenseven]

Terence Tao also uses Anthropic's models in his work. Oh, you didn't know that? Well, now you can pivot to saying that he's getting paid off by both companies. This is actually one of the hallmarks of both conspiratorial nonsense and military-grade cope. Any fact, regardless of how mundane or extraordinary, gets re-imagined as evidence of the same mad-hatter conclusion.

I hope people are screenshotting this stuff. This really needs to be documented. It's remarkable how wild it's getting.

[vanuatu]

kind of a hilarious conspiracy theory

You are correct that LLMs are trained on existing proofs but hiring researchers to solve unsolved problems is just unrealistic, both in terms of how none of the mathematicians simply came out and took credit for their own discovery or exposed this, and how training sets are not easily memorized (rather, the meta techniques are learned).

OpenAI just has better training methods and techniques for pure math over Anthropic, it’s one of their biggest strengths