Hacker News new | ask | show | jobs
by WhitneyLand 18 days ago
If all checks out this is a huge milestone. AI has now solved one of the most famous open problems in graph theory, using an off the shelf model, in one hour.

It might be a better mathematician than most humans at this point. Kind of like when chess software started beating everyone except grandmasters.

What’s left? Proposing and building out entirely new theories and frameworks? Then better than any human? Then alien math results we struggle to comprehend?

5 comments

>> What’s left?

For example, there's all the problems that the same off-the-shelf model hasn't solved despite OpenAI running it for many hours on them. Don't forget you're only seeing the results of successful runs.

We can estimate that those unsolved problems must number in the dozens, or even hundreds, given the amount of time that passed since the last announcement of a solution to an interesting problem by an OpenAI model: i.e. the unit distance problem which was announced solved in 20 May this year. That's a couple of months, yes? We can be fairly certain that OpenAI have been trying to solve other problems all this time, first because they are hell bent on demonstrating that their models can do maths and second because we just got another result, but it took that long. They were obviously not twiddling their thumbs all this time.

So if OpenAI are running their model on a single proble for eight hours at a time (according to the prompt they released) they could be easily have run a few hundred instances of their model on the same number of open problems 156 times for each instance (53 days since 20 May, with a model running in three eight-hour sessions per 24 hour day). I mean the only restriction is the cost they're willing to pay for the inference.

So yeah, there's a lot left to do still, don't worry.

The 2 most notable/interesting solutions have come from Open AI directly, but most of the 'LLM solves open problem' category didn't and has come from 3rd parties doing their own thing with publicly available models. I don't see why one would assume they're running models on hundreds of problems. Most likely they have a few problems they especially care about that they run on.
What, Erdős problems? It's hard to see how anyone except mathematicians, and then again only a few communities of mathematicians, especially care about those.

Remember back in the day when Deep Mind made AlphaGo? That made huge waves for two reasons: one, it was very well understood by AI researchers that beating expert humans at Go was very hard; and, two, that Go is a game of great cultural significance to literally billions of people outside of academia, albeit mainly in SE Asia.

Now, Erdős? I'm a computer scientist and I had to look up the planar unit distance problem when I heard about it because I had no idea what that was. Because it never comes up in the literature I read. I'm not saying it's not interesting, but it does seem a bit... random? That they started with Erdős problems? I'd have gone for a Millennium Prize problem, first. P vs NP, Riemann, Navier Stokes, those are heavy-weight results that would establish AI as the de facto approach to mathematics for the foreseeable future. Even I would find it hard to raise an objection (imagine that). Erdős can take a number, compared to all that.

I'm not saying they're choosing problems at random, mind. But it does seem like we're only seeing the tip of the iceberg, with respect to what the AI companies are doing internally. That shouldn't be a surprise. That's exactly how research works in general, both in academia and in industry. There is a clear survivorship bias and we only ever get to see the positive results, never the negative ones.

>> The 2 most notable/interesting solutions have come from Open AI directly, but most of the 'LLM solves open problem' category didn't and has come from 3rd parties doing their own thing with publicly available models. I don't see why one would assume they're running models on hundreds of problems.

Actually, that's a good point but it's in support of my contention. If there are random people in the community running LLMs on their own, favourite, maths problems, we should be seeing many more of those solved and much more often, provided LLMs were really as good at maths as OpenAI et al want them to be. There must be literally thousands of mathematicians trying to use LLMs to solve this or that problem that is famous in their community. Where are all those spectacular results?

Or, to abuse Fermi's question, where is everybody?

>What, Erdős problems? It's hard to see how anyone except mathematicians, and then again only a few communities of mathematicians, especially care about those.

Erdős problems vary enormously in difficulty and significance. The fact that a problem is obscure to non-mathematicians does not make its solution unimportant. There are also many major open problems in computer science that most laypeople have never heard of.

>That they started with Erdős problems? I'd have gone for a Millennium Prize problem, first. P vs NP, Riemann, Navier Stokes, those are heavy-weight results that would establish AI as the de facto approach to mathematics for the foreseeable future.

Solving the biggest, most famous open problems would "establish AI as the de facto approach to mathematics for the foreseeable future"? It would do a lot more than that.

>That's exactly how research works in general, both in academia and in industry.

Then what exactly is the objection? Research normally produces many failures and incremental results before a major success. Do you apply this survivorship-bias criticism to every published mathematical result, or only when a machine contributed to it?

>Actually, that's a good point but it's in support of my contention.

If they're running as many problems as constantly as you imagine then they did not miss all that. So either they're just not sharing it us which contends with your "desperate to demonstrate mathematical competence" or they have their eyes on a more curated set.

>Or, to abuse Fermi's question, where is everybody?

If we had as many verified alien encounters as LLM contributions to open problems, nobody would be invoking the Fermi Paradox. Where is everyone? Right here.

>> Then what exactly is the objection? Research normally produces many failures and incremental results before a major success. Do you apply this survivorship-bias criticism to every published mathematical result, or only when a machine contributed to it?

Yes I do. Not mathematical results specifically but generally research results. I might even have articulated that criticism on HN. I don't know if I could search for it easily though.

>> If they're running as many problems as constantly as you imagine then they did not miss all that. So either they're just not sharing it us which contends with your "desperate to demonstrate mathematical competence" or they have their eyes on a more curated set.

The people desperate to demonstrate mathematical competence are the AI companies. The people discussed in the part of my comment you quote are "random people" by which I meant the "3d parties" in your original comment.

>Yes I do. Not mathematical results specifically but generally research results. I might even have articulated that criticism on HN. I don't know if I could search for it easily though.

Okay Fair, but then this is a general research issue and not really a Open AI issue.

>The people desperate to demonstrate mathematical competence are the AI companies. The people discussed in the part of my comment you quote are "random people" by which I meant the "3d parties" in your original comment.

I'm not sure you got the point i was making. The point there was that if Open AI were running as many problems as frequently as you imagine they are then those 3rd party results should have been achieved by them, and if they're really so desperate to tell us how good the model is for math then why didn't they tell us ? Why have they only announced 2 results when they could have announced near a dozen by now ? Sure these 2 are in a class of their own, but some of the others are genuinely impressive in their own right and would certainly help that narrative you're talking about.

Either they're just not telling us and aren't as desperate as you imagine, or they're simply only interested/running models in a relatively few set of problems.

It's hard for me not to think what's the point. I am a very average, even below average person in times of intelligence. What is even my value or reason to be if I know anything I can do, LLMs can do better? What is even my value both on job market and as a human?
There are smarter and better humans at just about everything you or I could want to do, that's just life. Most of life isn't about comparative advantages, it's about enjoying life with people we like.
I can't enjoy life with my people if I can't eat, drink, and have a roof over my head.
Sorry to be nihilist, but you never had any objective value if you're thinking in these terms.

As far as we know, the universe "just is". There is no universal objective value of human beings, at all, any one of us.

You have to make or find your own value in the universe. I try not to think too hard about the nihilist side and try to appreciate that for some unfathomable reason, I seem to have what I call consciousness - the ability to observe the present and have it superimposed on the past, and what may be the future, leading me to "experience" things. I don't understand it, no-one does (some people suffering from the Dunning-Kruger effect think they do, but they don't), and yet, here we are.

So it doesn't matter to me if machines perform better than I do, because already lots of other people do. Just try to find your own joy or meaning, somehow.

> As far as we know, the universe "just is".

I don't know this. In fact, billions of people around the world don't know this. In fact, all evidence points to the contrary.

You have objective value being made in the image of a personal God. Denying that leads to a lot of pain, namely nihilistic suffering because it's on you to "pull yourself up by the bootstraps" in any endeavor involving your own self-worth.

Even Neanderthals? Were they also made in his image? We interbred with them. We can go all the way down the evolutionary tree here just say when to stop.
"In fact, all evidence points to the contrary."

Um. No? I'd like to see how you conjured up that one. Because evidence does indeed point exactly to things just being as they are.

"You have objective value being made in the image of a personal God."

Conjecture. Unsubstantiated.

"Denying that leads to a lot of pain, namely nihilistic suffering because it's on you to "pull yourself up by the bootstraps" in any endeavor involving your own self-worth."

No? Sounds like you're projecting your own belief here. "If I didn't believe in God, then what purpose is there to anything?" It's very small of a belief.

> made in the image of a personal God

How does that work? The physical image? Or the mental image? Either has big problems. An omniscient god is so cognitively beyond a mere "7+-2" human that the latter claim makes no sense at all.

This is one reason why I can't stand hn. Edgelord nihilistic comments that say nothing matters and take everything for granted get up-voted while credible and rational statements about God get down-voted.
That is an incredibly insulting comment. I am married, two children, have lead a fantastic fulfilling life. Just because I don't believe the Flying Spaghetti Monster created the universe doesn't mean I am an "Edgelord".

Remember the phrase, you also don't believe in God. There are hundreds of gods you don't believe in.

Bringing up the Flying Spaghetti Monster does make you an edgelord though.
What? What is rational about faith-based comments?

Literally nothing is rational about that. It's also just flat out off-topic.

To cite the famed Russian philosopher Norm Macdonald; "Scripture. Faith. Grace. Christ, Glory of God. Smart man says nothing is a miracle. I say everything is."
Do you have friends or people in your life that are also not geniuses? Do you think about them this way? Why or why not?
This is a great question. Thanks for asking it. It really got people talking.

My $0.02: Your value as a person never had anything to do with your value to the economy. It's time to relearn that fact.

Check out some of Tom Hodgkins books for more. Or maybe anxietyculture.com.

Yeah but what is a persons value economically anymore? What skillsets will enable us to continue to make a decent living till we are old?
It's funny because if people get replaced and don't have money they won't pay for API tokens either so the retail AI business collapses too, there won't be enough demand because people rather buy food.
But it is. My worth is based on what my value is, what I can do. What when I can no longer have any value on the job market?
Your economic value and your value as human being are two different things.

Yeah, you gotta eat, but some of us have been slaves for so long that we've started to believe that our self worth is defined by what we will fetch at auction. It is not.

Read more about the Protestant work ethic, it's history, and what life was like before men began to assume it was the one true view of life. Not very long ago nobody had clocks, and the idea that work was an ethic was a fringe, almost cult-like belief system. Now most humans routinely take stimulants every morning to be better servants, and mistake their job for their purpose on earth. That's just nonsense our culture invented though.

If the master wants you to do something, force him get out the whip. Don't let him live in your head rent free by adopting his crazy belief system.

YOU decide what your value is. Don't abdicate the decision to someone else.

Sorry, but most of us need to work to eat. This idea that our "value" is has nothing to do with the economy is an idea rooted in deep privilege, in the ability to say -- if I don't like my job, if I'm not employable, I can just retire, and the only problem will be figuring out how to live life afterwards.
AI will never be better than me at appreciating a good sunset.
Claude performs 50 microappreciations per second
Perhaps it will. And perhaps countless animals are already better than us at appreciating a good sunset, yet we do not seem to value them much.
Animals sure but they evolve with us for hundreds of millions of years so they get a break. They did their part.
You are a human being, one of the most wonderful thing the nature has ever created, besides all the other living beings and the wonderful earth we live in. Do not tell yourself you have a value just because some company may want to hire you or not.

Companies and industries already use tools and machinery for tasks were once done by human beings. AI is just another tool they will use and it will probably replace human beings from some intelligence related tasks.

However that may bring more disruption to the society if the government in your country do not protect and help people and leave free rein to capitalistic greed.

I'm my opinion that already happened in the US, not by using AI, but merely by using H1B visa to get intelligence worker from abroad. What happened is that the companies are doing great and getting the best smart people in the world but American people and society have been disrupted.

Live your life fully, be good to yourself and to others. Don't worry about the market.

Your value is intrinsic as a human being. We’re capable of love and shared experiences that a machine will never know.
You have inherent value by virtue of being human. Unfortunately it seems like people have forgotten humanism.
Remember that the most valuable human in the world is Elon Musk. What it means to be valuable is to be like Elon Musk. Calibrate your goals accordingly.
Manual labor
Military will always like warm bodies ;)

Sometimes it's easy for me to imagine a bad future like this; where most of the men are forced into military as the only org that still has use for wetware, and women are valued only for their ability to birth new people...

Anti-ai sentiment won't be automated out haha
Even that is being automated now, it was one of the thing being automated very early on in factories..
And yet we still have human laborers. Robots are a lot more expensive than AI inference.
Well, you provide training data to the savior and our Lord AI. /s
You don't compete with a chainsaw at cutting trees. You decide what the tree is for. Then sell the rainforest for shareholder value. https://youtu.be/UrgpZ0fUixs
And now AIs can do all that
> What's left?

I think humans will be left to propose new conjectures while machines fill out the proofs. I don't know if there are enough interesting conjectures to go round to build new careers, though.

To be able to propose interesting conjectures you need to walk the walk of trying to prove things yourself. That's not great for future generations.
It is the same with anything technical; you need taste and wisdom, which are borne of experience. I think society will want to subsidize this learning lest it deskills en masse.
But we can't have that experience when we deem the experience acquisition as sub-optimal to the grand scheme of things, which is to produce software.
It would be nice, but I don't have much hope as it doesn't help next quarter's profits.
Isn't it same argument for every tool we've created?
Surely the machines will have superior conjectures soon.
I think so. Mathematics is the least of my worries. I worry about what the machines will want to do.
You say those things like they're a short step away, but that might not be how it works out.

For example, AI has made zero progress in the last few years in surpassing professionals at art or writing. Its prompt-following skill is much better, and sure, it can render hands and text now, but its artistic sensibility is completely stagnant.

I think, and I may be totally off base, that the labs are specifically avoiding art and (non-technical) writing as an endpoint. It's bad PR for them- it calls attention to the copyright question and threatens the 'human flourishing' kind of jobs- and there's no money in it because people prefer art to be human made and there's hardly any money in that anyway.
The difference is that artistic sensibility is largely subjective. This means that:

1. It's hard to measure (and people can disagree about it)

2. It can't really be improved using RL without a human in the loop (which is how math is being trained)

At a certain level, yes, but AI is still so bad at writing that its failures are objective and easily measurable
I agree that AI writing is bloody awful, and that it's bad at creativity more generally, but are there actual objective measures of this?
Art is so entwined with the human condition that generative AI which conjures output from a statistical pass over background noise is bound to be stagnant.

Obviously it can impersonate art, but where creativity and the human story matter artists need not worry.

AI is no match for rapidly shifting goalposts ;)
AI-written stories have won several awards, e.g. https://lithub.com/a-prize-winning-story-published-in-granta...
What are you even talking about? The last few years, AI has made an insanely big jump in capabilities, performance, accuracy. It destroyed the carrier of a ton of writers and can generate images that are good enough to bamboozle people into thinking it's human made, that sounds like an insanely big leap to me, and yes it can be very creative and in music as well, I would bet it beats already 90% of musicians (most musicians are not that competent).