[cauch]

I find this kind of reasoning a bit pointless and unfalsifiable.

Someone says "LLMs fail at this", and you say "but humans also sometimes fail", then they says "but we are not talking about the same thing", and you answer "this difference does not matter because aliens may be totally different".

My point is that what we observe with LLMs does not require any understanding. And in some cases, it is clear the answer of a LLM was built without understanding. And in other cases, it looks like it could have been built with understanding because there is no visible errors, but because we know the LLM can build things without understanding, this can equally simply be something that is built without understanding and happen to have no error, and therefore just looks like it has been built with understanding.

I think you take the problem the wrong way: you start from the hypothesis that there is understanding, and then you are finding reasons to maintain this conclusion (the most prominent ones being "humans also can do mistake" or "... fake understanding" or "... hallucinate". Well, humans can do a lot of things that don't require intelligence, does it mean that things that do these things that do not require intelligence are in fact as intelligent as humans?). This is a confirmation bias.

I don't have problem if it turns out LLMs have understanding. But the reality right now is that a simple explanation is that it does not have it. But it feels like some people just argue "but it is still possible, bending this argument there and there". I bet at some point, they will say "ok, I see your point, but maybe LLMs are intelligent and have this behavior on purpose because they want to remain hidden because they are smart enough to understand that if humans would know, they would freak out". It feels more and more like a belief system rather than a scientific approach.

Just two elements to go further:

- in the majority of cases, "things that have been faked to look like there are the result of understanding" will be correct. Because if you are trained to pretend you understand, you are trained to imitate someone who has understood, and you are therefore trained to imitate their reasoning, which turns out to be correct. (if you want to test the understanding, it is complicated, because the "understanding" is a data leakage during training)

- if LLMs extract understanding for the data from their training, it is strange that their current understanding (just after the training) is so close to the current understanding of the humans. Surely humans have missed stuffs here and there. The math theorem number 3424 not solved yet is probably as "simple" than the math theorem number 6423 that happened to be solved by humans, it is just by chance and circumstances that some humans have worked on 6423 and found a way to crack it while they did not spend as much time and effort on 3424. And yet, LLMs just happen to never notice any theorem on their own at the end of the training phase. Asking a LLM "Explain to me a math theorem that humans did not notice, with demonstration. This theorem should be something you understood when you were trained over maths" just does not work.

(and, please, I know that a mathematician may know theorem 3424 and yet not have noticed 6423 either, or that LLMs can scan a math problem with a large series of math tools and find a break-through. But my point is that LLMs are studying math soooo intensively during the training that they know all the human theorems, which is way more knowledge than any single mathematicians. And yet, it turns out that all this math understanding just ends up being exactly limited to what humans already know. What are the odds? Or more probably: they just don't really understand math, and when asked about a known theorem, they generate a correct explanation based on training data without properly understanding it)

[dagss]

I didn't really try to argue that LLMs "have understanding".

I am more, in a sense, arguing that "understanding" isn't clearly defined and sceptical about your confidence that this is an obvious quality of humans.

I'm not getting what this "understanding" thing is in humans that you are talking about. But I feel if anyone you are the one making the unfalsifiable statements here. You are the one talking about an inner quality within the reasoning that only humans possess and not LLMs.

If I re-read your post and replace the word "understanding" with "consciousness" then it makes a lot more sense to me. Yes, humans can be conscious that they understand something, while LLMs are very very probably not conscious of anything at all. If that is what you mean, I can easily agree with that. I would never argue LLMs are conscious.

But at least my original post had nothing to do with consciousness.

If I'm coding, I'd definitely pick the "understanding" of Fable over a junior engineer's "understanding" any day, for purely pragmatic reasons. When I say that, I simply mean that the rate of mistakes in junior humans is way higher than in best trained LLMs for most coding tasks.

I'm guessing this is not using the word "understanding" in a way you are happy with, and probably because you define the word "understanding" as being related to consciousness?

[cauch]

> I am more, in a sense, arguing that "understanding" isn't clearly defined and sceptical about your confidence that this is an obvious quality of humans.

I did not do that, because it is not what I believe. I don't have any problems with the concept of having non-human being intelligent. It is just that LLMs are not that.

> I'm not getting what this "understanding" thing is in humans that you are talking about

Then that's fine. Other people have a better understanding of the notion. Just simply avoid the conversation if you don't know, it just feels like you are muddying it by not getting the concepts.

> If I re-read your post and replace the word "understanding" with "consciousness" ...

No, I'm not talking about "consciousness". I'm talking about "perceiving the underlying meaning or concept". LLMs don't create their answers by relying on the abstract concepts of the objects they are using, they just have meaningless rules linking the different objects, without grasping the abstract concepts explaining these links.

> I'd definitely pick the "understanding" of Fable over a junior engineer's "understanding" any day

Similarly, I trust better my pocket calculator than a human, but it does not mean that the calculator "understand math", it just has the "math rules" hardcoded without grasping the abstract concepts. In LLMs, the rules are not hardcoded, just extracted from the data, but the LLM doesn't understand any more than a pocket calculator understand math.

[dagss]

Thank you that was clearer.

So, I have a PhD in Astrophysics so I am not a total stranger to doing some thinking. And I would say "create (...) answers by relying on the abstract concepts of the objects they are using" is a lofty goal for humans, something to aspire to more than something that typically goes on. We go by habits and intuition and allegories and quite muddy concepts most of the time. Concepts are malleable and evolve in clarity. And in creating new mathematics etc., intuition, inspiration, "flashes of insights" etc after absorbing oneself in the problem has an important role.

Are these things we have in our minds, whether concepts or habits or intuitions or flashes of insights, better or worse than whatever patterns could potentially be found in the LLM weights?

I struggle to label one of them "understanding" and the other not, at least without involving consciousness somehow.

Obviously you can define "understand" as "understand as a human would" but that is circular and uninteresting.

We just have to agree to find each others position incredible I am afraid :)

[cauch]

> ... is a lofty goal for humans, ... We go by habits and intuition and allegories and quite muddy concepts most of the time

Those are already concepts. For LLMs, the word X is just an object linked to the words W, Y, Z, with no meaning to it. Habits, intuition and allegories are using objects to which we are attributing meaning.

Just to clarify, the links that LLMs create are complex, for example depend on all the surrounding other words, but they are still meaningless. If 2 totally different semantic sets of words happen to have exactly the same graphs of links, then you can swap the sets of word together, it does not matter for the LLM. To use a simplified example where you reduce a set to just 2 words, if "garden" and "pea" are linked the same way that "quantum" and "mechanic", then the relationship are the same for the LLM, without the LLM understanding that "garden pea" is a different concept than "quantum mechanic".

That's what I mean by "understanding": humans understand "garden pea" and "quantum mechanic" as concepts (even if they don't know biology or physics enough to even explain how they work), LLMs just use these objects as meaningless entities. All there is is a graph of links used to generate a sentence, but without knowing what the sentence means. A bit like if someone was giving you all the words of a language you don't know and the exact rules of how to build an answer given an input, but that you don't know what each word means.

Of course, the relationship learnt by the LLMs are very complex, allowing big changes based on the surrounding other 100'000 words. But learning this relationship is still more straightforward than to leap into a conceptual world model (especially because there is nothing to guide the LLM. If "garden pea" and "quantum mechanic" have the same graph geometry, then the world model where "garden pea" is an abstract field of study and "quantum mechanic" is a material object is as probable than the opposite).