[MrDrDr]

> "Even though I can motivate it in retrospect, ChatGPT’s idea to use h^2-dissociated sets to control relations of order at most h feels quite ingenious. As far as I can tell, this idea is completely original."

The question that keep bothering me is can an LLM generate an idea that is truly novel? How would/could that actually happen? But then that leads to the question - what are we actually doing when we think?

Perhaps it's as simple as the ability to just make mistakes that matters, the same things that powers evolution. As long as the LLM can make mistakes, it's capable of generating something genuinely novel. And it can make more mistakes much faster than we can.

[charleshn]

Yes, they can.

Some people like to parrot "next token prediction", "LLMs can only interpolate", and other nonsense, but it is obviously not true for many reasons, in particular since we introduced RL.

Humans do not have the monopoly on generating novel ideas, modern AI models using post training, RL etc can come to them in the same way we do, exploration.

See also verifier's law [0]: "The ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI."

This applied to chess, go, strategy games, and we can now see it applying to mathematics, algorithmic problems, etc.

It is incredibly humbling to see AI outperform humans at creative cognitive tasks, and realise that the bitter lesson [1] applies so generally, but here we are.

[0] https://www.jasonwei.net/blog/asymmetry-of-verification-and-...

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[vld_chk]

I genuinely start to think that we, as humanity, severely overestimate our cognitive abilities. We act so surprised “just a few years of LLM with a few RL tweaks match our PhD levels! It must be hidden inside our knowledge base!”. Em, what if no? What if our “PhD level” is just very low level comparing to upper boundaries of measurable intelligence? What if we need to learn being humble and stop treating our minds as “sacred source of creativity and intelligence”?

[energy123]

RL or no RL, AI cannot escape the distribution it's trained on. It's just that the labs will put so much into the distribution that we won't be able to tell the difference that easily, nor will it matter for most tasks. The reason AI does well on ARC-AGI-2 is because the labs created synthetic training data using similar puzzles.

[crthpl]

Yes it can! That's the whole point of RL! it generates slightly out of distribution rollouts, and rewards good rollouts to change the distribution of the output

[energy123]

That's not out of distributíon, that's inside the distribution of the rollout. If you don't create rollouts for the game of Chess then it doesn't know how to play Chess no matter how smart it is at tasks you've created rollouts for. It's structurally stuck in its distribution.

[djeastm]

What if it doesn't need to escape the distribution, it can just exhaust the current distribution we have much more broadly and efficiently than humans can?

So the answers we're seeking to our bleeding edge questions are already there, we just need an AI's ability to target the answers. Then re-train on the improvements and go from there.

Just a thought.

[jdub]

Reinforcement learning for "reasoning" perturbs the model to generate completions in a particular chain of thought / alternative selection structure. It's three next token predictors in a trench coat.

[munksbeer]

When these things start solving many more long standing problems, and start introducing more novel problems, will people finally admit that the "next token predictor" is not the gotcha they think it is?

[jdub]

It's not a gotcha. It's incredible what these things can do despite being next token predictors from a weird dataset. That's at the heart of the "bitter lesson", and you don't have to believe in magic to see it.

[charleshn]

> Some people like to parrot "next token prediction", "LLMs can only interpolate", and other nonsense

Thank you for illustrating my point.

[eterm]

My own take, and it's veering into the Philosophy of Mathematics, but there's a debate about whether Mathematics is "Invented" or "Discovered".

If it's "invented", then it requires ingenuity.

If it's "discovered", then it was always already there, just waiting for the right connections to be made for it to be uncovered and represented in a way we can understand.

Invention requires ingenuity, but discovery does not. So if LLMs can generate truly novel mathematics, for me that settles it that mathematics is indeed discovered, as LLMs are quite capable of discovery yet I don't consider them possible of invention.

[layer8]

Mathematical concepts are invented, but they live in a space of possible (conceivable) mathematical concepts, and we can only invent concepts from that selection of possible concepts. This can be reframed as a process of discovery regarding which conceptions are possible.

Furthermore, the results of theorems aren’t an invention, they are a discovery of what the base assumptions (axioms) logically entail. Finding out which theorems are true and provable is a discovery process. For example, the results of Gödel’s incompleteness theorems were a discovery. They weren’t invented, in the sense that the results couldn’t have been otherwise. We merely could have failed to discover them.

This also holds for physical inventions. You discover a working way to build some functioning mechanism. It’s a process of discovery of what is possible in the physical world.

Whether you portray somethings as a discovery or as an invention is more a matter of degree, a matter of from which angle one is looking at it.

The possible states of an LLM are finitely enumerable. The same likely holds for the possible states and configurations of a human brain, in approximation. Therefore there is only a finite set of possible ideas, thoughts, and conceptualizations an LLM or a human can have, and in principle they could be exhaustively enumerated and thus “discovered”.

[MrDrDr]

I like this distinction, but it would then seem the only 'invention' would be the axioms of your mathematics. There exists numbers (natural, imaginary...), there exist shapes (a point, a line...). All the work from that point on could be 'discovered'. I agree that I don't see LLMs inventing in this way. But again, it raised the question - what are our brains doing when we 'invent' something?

[strgcmc]

Well, take any invention you like, and let's break it down.

Somebody at some point, "invented" the idea that the earth was round. Before that, the obvious "just look around you" answer would've been, duh of course the ground is flat. But we know the earth has always been round, even if humans couldn't appreciate it for hundreds of thousands of years (I don't count the pre-history before homo sapiens). So we "invented" some fields of science and the mental models / abstractions that allowed us to conceptualize what a round earth could mean and how to measure it, but we didn't invent the roundness itself -- that was always reality, and we just lacked both the thoughts and the tools to conceptualize it (until later).

Now you might say, well that is a category of "simple" physical observations. The earth is naturally round all the time and doesn't take any extra human effort to make it so (it took some effort to imagine that it could be and to find ways to measure/prove it). But what about say -- semiconductors, NVIDIA GPUs, that sort of thing? It's not like semiconductors grow on trees and we just need to find them and learn how to consume/use them... isn't that a better example of "true invention"?

Sure, I could see that. But I guess my POV would be that, the invention of the latest AI chip, or the first semiconductor, or the first vacuum tube, or whatever came before, all laddered largely incrementally on "discoveries" that were then cleverly tweaked or reapplied, so that what appears to be "true invention" is usually/more-often just another chain in a long chain of "discoveries" that led up to it. I grant you that some of what appears in hindsight to be continuous progress, really is built on small discontinuous "leaps", but I don't think that breaks the argument (strengthens it in fact, IMO). You wouldn't have semiconductors today, unless Faraday (or somebody like him) discovered that silver sulfide resistance decreases with heat, and that is more like one of those physical properties that reality has always had (much like, earth was always round, we just didn't know it at first).

So in that sense, I feel this becomes almost like an "evolution vs intelligent design" debate -- some people look at the complexity and miracle that is the human eye or the human brain, and they insist there must have been an intelligent designer, because surely no random chaotic biological process could have produced something so wonderful... And yet, I think the scientific evidence largely shows that, indeed that is what happened, just random chance + evolutionary-pressure was all you really need (plus billions of years). So if you can accept that analogical framing for a minute, then I would posit that "invention"-adherents are really making something like an intelligent design argument, vs "discovery"-adherents are saying that evolution (in an artificial sense, with the artificial selection pressures of scientific research, of capitalism, etc., and compressed into centuries or decades, not millions or billions of years) is sufficient to derive miraculous-seeming results. The little discontinuous leaps along the way, are kind of like the random mutations of genes that happen to confer an advantage -- maybe we can say that we are more intentional about seeking those leaps out, or maybe we are just right-place/right-time lucky (e.g. thinking about penicillin and the random petri dish left out).

Perhaps once (or if) there is the sort of leap that breaks us out from a Type I to a Type II+ Kardashev civilization, maybe then I would grant you something needed to be "invented" that couldn't be based on a line of "discoveries". Or maybe not, maybe it will just be another semi-random discovery.

[eiieue]

Mathematical objects are an invention of the mind - they are abstract objects that only an entity who can process abstractions can make sense of.

There is no ‘discovery’ here nor was it waiting to be found. The human has to sacrifice and pursue the path of exploring reality and thereby is inherently inventing.

Humans built up mathematics iteratively from smaller bases extending into large ones. Is this what LLM’s do? Of course not - They are fed with vast amounts of information from the off.

[LiamPowell]

Trivially the answer is yes by the infinite monkey theorem. If we allow the sampler to pick any token then any stream of arbitrary tokens can be generated. Therefore if an original idea can be represented with written words then a LLM can generate it. That is perhaps not the most satisfying answer, but if you want a better one you'll need to provide a function that determines if an idea is original.

[humanfromearth9]

For my paper about ME/CFS, I let an LLM integrate lots of findings of other scientific papers. Then I ask the LLM to "creatively brainstorm", given all we know of ME/CFS and the newly integrated paper, to generate new hypotheses, treatment ideas or any other kind of insight it can think of.

This works really well.

Now, it's clear that I have no idea how much of this is something we would consider new and original, and how much is a kind of systematic, but not novel, easy of thinking.

What I couldn't do so far is get an LLM to generate a truly new maths theory, with new abstract concepts and dimensions and points of view. The kind that is not just a combination of existing theories and logic.

[realtalk_sp]

Would you mind posting the outcome of this? A person I love dearly is struggling with Long Covid/CFS. I’ve been doing something similar to what you describe, but I’m always looking for more angles that could help.

[jasfi]

It's about the ability to combine ideas in novel ways, without breaking the rules in relevant frameworks. Sometimes the idea may even be to contradict existing theories where they are weak.

[ikari_pl]

How do you define a new idea?

To me, it's rearranging the information you had in a way that hasn't been applied or published before.

That's literally what LLMs are built for.

[eiieue]

Theres a simple test for this.

Limit the knowledge an llm to some point in time at which a discovery was made. And check to see if the llm could produce the discovery.

If you think OAI hasn’t already tried this then think again - they have every incentive to do so and announce it to the world.