[mohsen1]

It seems like we forget that LLMs are next token prediction systems. Using raw models without instruction following and chat completion bells and whistles will give you a better feeling of what LLMs are.

The current interface to LLMs are heavily biased towards "predict the next token in the context of a user with a helpful assistant" but LLMs are capable of other modes of next token prediction too.

Before the ChatGPT release people often measured LLM performance by how well they could produce a coherent story or a poem. that's where Anthropic model names are originating from I am guessing.

[jhbadger]

The problem with that argument is it is trivial to write a Markov chain program that takes in text and then can generate the most probable series of words given a starting word. I myself wrote such a program in BASIC on a 64K 8-bit computer in the 1980s after reading one of A.K. Dewdney's columns. That wasn't at all an LLM though. There's a connection, sure, but one that is equating a paper airplane to a jet airliner.

[Charon77]

The issue with Markov Chain is you can't get good next token prediction on long enough context because once you see the last 1000 words instead of just 2, it's quite unlikely that your 'frequency' is populated for that exact combination, and markov chain don't work on token embedding that allows some encoding of meaning.

[AlecSchueler]

> and markov chain don't work on token embedding that allows some encoding of meaning.

Working on an "encoding of meaning" sure sounds a lot like reasoning.

[tanseydavid]

Consider the classic ambiguity example from Natural Language Processing: "Time flies like an arrow; fruit flies like a banana".

Markov chains are mostly about memorized local co-occurrence, while transformers are much better at context-sensitive disambiguation and compositional structure.

I do not have deep knowledge about these subjects but this is what I think was meant by "encoding of meaning" (LLMs). I also do not think this capability has to be thought of as "reasoning".

[post-it]

> It seems like we forget that LLMs are next token prediction systems.

It's pretty clear to me that above a certain size threshold, LLMs are more than a sum of their parts. The sheer amount of training data seems to embed a higher level of reasoning.

[geon]

No. The reasoning is just an illusion. As soon as you look at it closer, it falls apart completely.

https://yosefk.com/blog/llms-arent-world-models.html

https://www.reddit.com/r/programming/comments/1mnc9qf/llms_a...

https://news.ycombinator.com/item?id=44854518

[post-it]

This is an excellent comment from your HN link:

> The core issue there isn't that the LLM isn't building internal models to represent its world, it's that its world is limited to tokens. Anything not represented in tokens, or token relationships, can't be modeled by the LLM, by definition.

I've noticed that LLMs struggle with tic tac toe, even when I get them to draw it out. And it makes sense! Tic tac toe is an extremely visual game, and difficult to reason about in a single dimension.

[solid_fuel]

> It's pretty clear to me that above a certain size threshold, LLMs are more than a sum of their parts. The sheer amount of training data seems to embed a higher level of reasoning.

You're staring at a mirror.

I have never seen any compelling evidence that LLMs actually reason. They are good at predicting the next token, and that is all they do. This action may be useful, but it is not reasoning.

[gmueckl]

There cannot be any reasoning embedded in the model. The algorithm is literally "predict the most likely next token". Anything beyond that is just patterns in the predictions fooling us humans into ascribing more to the system than it is actually producing.

[wizzwizz4]

LLMs are great big finite state machines. Finite state machines can perform mechanical reasoning. A priori, there can be reasoning embedded in the model. I agree that (these) LLMs don't generally reason (even when they're writing words like "I reason that, since X, we have Y, therefore Z"), but that's not because a model inherently cannot do that.

[linzhangrun]

Quantity change leads to quality change. You can check out this Kurzgesagt video on emergence: https://www.youtube.com/watch?v=16W7c0mb-rE

[thepasch]

> The algorithm is literally "predict the most likely next token".

That's confusing the training objective with the learned behavior. It's like saying "Stockfish's algorithm is literally 'minimize this number', and therefore, it can't actually play Chess."

[gmueckl]

Not a valid comparison. Chess algorithms are built around the rules of chess, most notably the turn taking nature of the game (min/max with alpha/beta pruning based on lists of valid moves in any position).

[etskinner]

Who's to say the "rules of reasoning" aren't just predicting the next thing that an intelligent person (you) would do? Emergent behavior isn't magic, it's just emergent.

[aspenmartin]

This is just a misconception of how LLMs work and also what reasoning is.

“There cannot be any reasoning embedded in the model” a strong statement, what do you mean by reasoning because by any reasonable definition I’m aware of, they clearly are able to exhibit reasoning.

The fact that the pre training objective is next token loss has nothing to do with capabilities or their ability to reason. To be highly successful at next token prediction you NEED to reason. I’m quite confused here.

[gmueckl]

LLM output produces the illusion of reasoning. The underlying computation, however, is not reasoning.

[aspenmartin]

If you don’t mind actually taking a few more words to be more specific that would be helpful because what you’re saying doesn’t really make sense at all. You don’t need to trust that the reasoning traces are all faithful representation of an internal reasoning trace. Plenty of other ways to probe models (see anthropics work using circuit tracing).

[gmueckl]

What else is there to say? LLMs can at most regurgitate approximations of human reasoning steps in the limited forms in which they may be expressed in the training data or interpolations thereof. That's the core essence of what they are. There is no proper reasoning to be found.

[thepasch]

How do you define reasoning? What does a system have to functionally do in order to qualify for it?

[gmueckl]

Reasoning includes things like proper use of logic. LLMs have been repeatedly shown to fail horribly at this.

They consistently fail at drawing basic logical conclusions because they cannot build a sufficiently abstract model of certain problems that allows them to grasp their true nature. In other words, the whole class of questions of the kind of "how many r's in strawberry" or "do I take the car to the car wash?" would be answered correctly and reliably.

[applicative]

LLM output is a kind of dreaming but with the whole of past human text output as dream material. It turns out to be useful if you can direct the hallucination