[cornholio]

I think it's becoming harder and harder to argue that LLMs don't really reason and just mimicry human speech. This counterexample is clearly the result of a sequence of steps that build on previous knowledge in context and logically combine it to reach other true statements - to a degree and complexity that rivals the best human minds.

For someone that use Claude Code every day, this is obvious, but for some reason many scientists refuse to accept that it's truly reasoning; perhaps not in the human sense, but in a very profound and real sense. These powerful results are devastating to their point of view.

I can sympathize, because I too called LLMs "fancy Markov chains" in the GPT 3 era. But there comes a time where you have to update your world view to match reality, or be stranded in fantasy land.

[TeriyakiBomb]

I disagree. It's still extremely possible to "google whack" an LLM on a topic with little publicly available information. If you ask questions about APIs in desktop software, say something like Houdini, it'll produce a bunch of non-sequiturs. A lot of that knowledge lives offline in VFX studios, but it's just some fairly run of the mill python API stuff. You'd expect it to do better.

Similarly I've run into several "you just gotta know" type problems where LLMs still just fail. My favourite was a quirk in how async relationships work inside emberjs. Three different models gave a variation of the same incorrect answer. When I searched myself, I initially came up with nothing and eventually found what I think might be the only example of the same issue on the internet, a single stack overflow question wth two responses. The first is what Claude, gemini and chatGPT said, the second response was the OP saying it was wrong. I asked in the Ember discord and a core team member responded instantly with the answer.

There's a video I love on Youtube, where a guy uses ML to assemble a blank jigsaw. It performs amazingly, the jigsaw being blank is of no consequence and if it did have an image, it'd perform worse. That's all LLMs do. Just because the jigsaw pieces are smaller, they're still just getting assembled in whatever way fits, there's no mechanic for interpreting the image on the front.

[abernard1]

I think this is actually a perfect case of an impressive thing being done exactly by mimicry.

Why do LLMs still have trouble on floating point math without forking out to a tool, but they can perform symbolic manipulation just fine?

Because symbolic manipulation is just rote work and textual stepping through symbols. A side poster commented on the number of prior attempts on this problem which were close, but not quite.

Starting from a known "close" solution (which this did), and using exploration to search around the space is exactly something an LLM would and could be good at (clearly).

The "transformer LLMs are next-token predictors with some in-GPU processing of bounded complexity with respect to token count" remains undefeated. Both because that is mathematically what they are, and also because we don't have counterexamples to that effect that don't require some higher-order tooling wrapping the systems.

Symbol manipulation is what LLMs are good at.

[cornholio]

The field now favors into the view that symbolic manipulation is not the mechanism of general intelligence, but rather an emergent byproduct of learning. So the fact that a connectionist machine (neural network) got so good at symbolic manipulation actually supports the view that we are closing the gap to general intelligence. Through the rote work, the machine really internalizes those rules and the symbolic manipulation capabilities are emergent, just like we humans do it.

What still confuses people is the insane inefficiency of deep learning, and that those emergent capabilities require such an immense training corpus compared to the only other architecture that we know of.

But this already is an optimization problem. If the machine gets super human at symbolic reasoning, and at the same time, can solve the symbol grounding problem to real world data and sensors, what prevents you from saying it thinks? Can it not solve real world problems? Can it not redefine its tasks and display some form moral agency - even if a totally foreign morality for us humans? Can it not use these abilities to reproduce and expand, create ships and turn the universe into paperclips, if it finds it worthwhile?

Math is basically just a playground that is perfectly suited for these emergent capabilities, so of course we will see the first progress here; but there is no firewall separating math problems from general cognition.

[abernard1]

I am not confused. Because herein these forums, I predicted everything that was going to happen years ago.

And the "insane inefficiency" of deep learning is fully to be expected from how it works. As well, there are provably no—literally no—emergent properties in these models. The choice of metric was a convenient, sloppy, and embarrassing fault of the field. It should be discredited; the field should be embarrassed; expectations on messaging should have changed; and it did not.

Why? Because the industry is full of charlatans, and this is a highly profitable enterprise telling people that this would lead to AGI.

Multiply two floating point numbers without a tool call. Still can't, because it's a curve fit.

So, in summary, nothing you just said is relevant. There are no emergent capabilities, simply 1) search, + 2) the original set of learned feature vectors from throwing tons of data at this.

[snarkconjecture]

Current LLMs can absolutely multiply floats without a tool call. In fact, that's a much more rote symbol-manipulation task than doing original math research.

[abernard1]

With what accuracy? And with how many intermediate tokens?

We can replace multiplication with any class of problems which should go from 0->100% solution almost immediately if there was actually a concept learned.

There is not. Because they are plain ol' fits. And there are no "emergent" features that pop out without having a sufficient set, where "sufficient" is absolutely gigantic and equivalent to memorizing enough of the space to compress the problem. LLMs are Rain Man.

They interpolate within a known distribution. Search allows places outside of distribution to be explored.

This paper should be required reading [1]. You can explore the curves yourself. You can see exactly what it's doing. And you also have this nagging thing—which you know and I know—that all these models converge and do not diverge upwards. An "emergent" "hyperintelligence"—a characteristic that could be found if something was actually learned and combined with a new concept—would not have this problem.

Exponentials on exponentials added to compute and data and the problem classes still sit at not great places, and require agents, feedback loops, and trial and error to solve. The models are the problem, but more importantly, the people selling things these models could never do are the problem.

[1] https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...

Edit: It should be mentioned, if there's some scary neural architecture that's super-de-duper and doing something beyond the very obvious next string prediction that LLMs clearly do, it can't do what absolutely ancient ML models could do; a network to multiply two floating point numbers should pop out somewhere without symbolic computation, no?

It does not. There's no magic other than the run-of-the-mill SV fake it til you make it magic. And that magic has failed.

[cornholio]

> With what accuracy? And with how many intermediate tokens?

So you are arguing here that LLMs should just "know" the result of a multiplication when the operands are in context, ie, that a hidden multiplier circuit should emerge in their weights.

Is this how you do multiplication, if I give you two 12 digit numbers, does the 24 digit multiplication result just pop in your head? Don't you have to follow a learned algorithm through a tedious system 2 effort? Don't you need to write down the results on paper because you can't actually hold in your head the dozen partial results, each with a dozen digits? How many visual, tactile and reasoning tokens does this consume, moving the hundreds of muscles that make up your hand to draw each number under visual feedback, then reading all those numbers back and transforming ocular activation data into numeric symbols?

It seems to me your system 2 is just following a symbolic algorithm for multiplication, and it does roughly the same steps as the LLM trace I showed you previously.

So if you think this is the mark of your intelligence, why wouldn't it apply to the machine too? Why is it implausible that, following a similar algorithm learned from some mathematical paper, the LLMs has reasoned a new solution to a problem in another math field? Why couldn't the machine combine and morph these algorithms for symbolic manipulation, to yield entirely new and original results? How would those results differ from results human mathematicians generate, using recipes they learned in university?

The emergent behavior that we talk about isn't that the machine can do multiplication in its "head" after knowing the multiplication algorithm. What emerges is the ability to follow any other algorithm, even algorithms that were not in the training set, even algorithms to create other algorithms, which it then executes. This is the emergent behavior that matters for AGI; once it can do that, it's a trivial exercise to create a non-AI tool to automate and accelerate the mechanical tasks - just like we humans do it.

[pickleRick243]

"I am not confused. Because herein these forums, I predicted everything that was going to happen years ago."

I mean, lol.