[PaulDavisThe1st]

I prefer to work the other way around. That is, accept that a lot of human speech (and text) is generated via similar mechanisms to the ones that drive LLMs, but note that there is another kind of behavior - reasoning - which seems to be distinct.

[naasking]

I think you need understanding to reason, but you don't need reasoning to understand. A child understands how to catch a ball without reasoning about forces, air resistance, gravity, etc.

I think LLMs understand without reasoning. They've built a large associative network of concepts (a kind of understanding), but we don't yet have a good handle on the process of reasoning using that network.

[PaulDavisThe1st]

I don't think it is useful to say that a child "understands how to catch a ball", even though it is something many of us do say quite often.

The child knows how to catch the ball, without understanding. Later, the child learns both reason and physics, and can reason about ball catching in a different way.

I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.

[naasking]

> The child knows how to catch the ball, without understanding.

I disagree that decoupling "knowing" from "understanding" is well motivated or useful in any way.

> I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.

This is starting to seem like a Chinese Room disagreement, so I'll just say that what that thought experiment demonstrated is that there is no robust or meaningful way to distinguish semantics from syntactic processing given everything we know about reality, and that LLMs were a fantastic demonstration of this illusion, eg. that "semantics" are reducible to a relational network of syntactic elements.