[dns_snek]

> It’s not an incorrect model of the world as technically both you and an LLM ultimately have an incorrect model of the world and both you and the LLM fake it.

I should've said that the model is "missing", not "weak" when talking about LLMs, that was my mistake. Yes I'm a human with an imperfect and in many aspects incorrect conceptual model of the world, that is true. The following aren't real examples, they're hyperbolic to better illustrate the category of errors I'm talking about.

If someone asks me "can I stare into the sun without eye protection", my answer isn't going to change based on how the question is phrased because I conceptually understand that the radiation coming from the sun (and more broadly, intense visible radiation emitted from any source) causes irreversible damage to your eyes, which is a fact stored in my conceptual understanding of the world.

However LLMs will flip flop based on tone and phrasing of your question. Asked normally, they will warn you about the dangers of staring into the sun, but if your question hints at disbelief, they might reply "No you're right, staring into the sun isn't that bad".

I also know that mirrors reflect light, which allows me to intuitively understand that staring at the sun through a mirror is dangerous without being explicitly taught that fact.

If you ask an LLM whether staring into a mirror which is pointed at the sun (oriented such that you see the sun through the mirror) is safe, they might agree that it's safe to do so, even though they "know" that staring into the sun is dangerous, and they "know" that mirrors reflect light. Presumably this is because their training data doesn't explicitly state that staring at a mirror is dangerous.

The way the question is framed can completely change their answer which betrays their lack of conceptual understanding. Those are distinctly different problems. You might say that humans do this too, but we don't call that intelligent behavior, and we tend to have a low opinion of those who exhibit this behavior often.

[ninetyninenine]

No it doesn’t. Conceptual understanding is there. But the LLM is not obligated towards correctness. The fact that at one point it gave you the correct answer is indicative that an aspect of it understands the concept.

Like if I told it solve a complex puzzle equation not in its training data and it correctly solved that problem. We know from the low probability of arriving at that solution from random chance that the LLM must know and understand and reason to arrive at that solution.

Now you’re saying you perturb the input with some grammar changes but leave everything else the same and the LLM will now produce a wrong answer. But this doesn’t change the fact that it was able to get the right answer.

Humans can be dumb and inconsistent. LLMs can be dumb and inconsistent too. This happens to be a quirk of the LLM. But you cannot deny that it is intelligent on the sole fact that LLMs can produce output that we know for sure can only be arrived at through reasoning.

[dns_snek]

> The fact that at one point it gave you the correct answer is indicative that an aspect of it understands the concept.

Having a conceptual understanding means that you always provide the same answer to a conceptually equivalent question. Producing the wrong answer when a question is rephrased is indicative of rote memorization.

The fact that it provided the right answer at one point is only indicative of memorization, not understanding which is precisely the difference between sometimes getting it right and always getting it right.

[ninetyninenine]

>Having a conceptual understanding means that you always provide the same answer to a conceptually equivalent question. Producing the wrong answer when a question is rephrased is indicative of rote memorization.

False. I can lie right? I can shift. I don't need to be consistent. And I don't need to consistently understand something. I can understand something right now and suddenly not understand later. This FITS the definition of understanding a concept.

But If I gave an answer that has such a low probability of being correct, and the answer is correct, then the answer arrived at by random chance. If the answer wasn't arrived at by random chance it must be reasoning AND understanding.

The logic is inescapable.

[dns_snek]

> I can understand something right now and suddenly not understand later. This FITS the definition of understanding a concept.

Not any definition that I would agree with, that's for sure.

[ninetyninenine]

You must agree with it. The fact I can formulate a sentence with it indicates it fits with the colloquial definition of the word. Every human recognizes it, even you. You’re just being stubborn.

When I say I can understand something now and then not understand something later it doesn’t violate the definition of the word. Now you are making a claim that your personal definition of understanding is violated but that’s also a lie. It’s highly unlikely.

First of all death. I understand something now. Then I die, I don’t understand something later due to loss of consciousness.

Amnesia. I understand something now and I don’t understand something later due to loss of memory.

In both cases someone understood something now and didn’t later. Every human understands this conceptually. Don’t lie to my face and say you don’t agree with the definition. This is fundamental.

The act of understanding something now and then not understanding something later exists as not only some virtual construct by human language but it exists in REALITY.

What happened here is that when I pointed out the nuances of the logic you were too stubborn to reformulate your conclusion. It’s typical human behavior. Instead you are unconsciously re-scaffolding the rationale in order to fit your pre existing idea.

If you’re capable of thinking deeper you’ll be able to see what I’m in essence talking about this:

In the gap between prompt and response. The LLM is capable of understanding the prompt and capable of reasoning about the prompt. It does so on an ephemeral and momentary basis. We can’t control when it will do it and that’s the major issue. But it does do it often enough that we know the LLM has reasoning capabilities however rudimentary and inconsistent because the answer it arrives at via the prompt is too low probability to be arrived at using any other means OTHER than reasoning.