[mjburgess]

The "failure modes" in humans do not show we lack the capacity.

Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object.

> I've found more often than not, simply changing names of variables

Yes, lol --- why do you think that is?

Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case.

P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference)

To such a degree that the latter is vanishingly close to zero.

[famouswaffles]

>The "failure modes" in humans do not show we lack the capacity.

Then they don't in LLMs too

>Yes, lol --- why do you think that is?

Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.

[mjburgess]

> Then they don't in LLMs too

LLMs don't get drunk .

If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not.

They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.).

Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain.

But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics.

In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding.

Cognitive impairment = reasoning machine is broken

Environmental illusion = data is ambigious and actions cannto resolve it

These "failure modes" are expected if you actually have the relevant capacity.

[moffkalast]

> LLMs don't get drunk .

Well actually they sort of can...

https://www.reddit.com/r/LocalLLaMA/comments/13vv941/tempera...

[famouswaffles]

>Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain.

alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?

[mjburgess]

see, just one reference in the paper: https://arxiv.org/pdf/2302.08399.pdf

[famouswaffles]

they can answer those

https://medium.com/@nathanbos/prompting-better-theory-of-min...

[mjburgess]

Yes, by changing the words

The whole point is that irrelevant word permutation should not "turn on" or "turn off" this capacity.

That you can "prompt engineer" your way to the answer shows that the prompt engineer knows the answer and can "use the right search terms" to find it.

[YeGoblynQueenne]

>> alright let me humor you for a bit.

That's real class, right there.

[lostmsu]

> less so. But not if I permute the name of the object.

You need to realize that you wrote it on a forum where the most known joke is "there are two hard things in programming". That would immediately show you how this assumption is exactly false.

[lgessler]

This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define that.

[mjburgess]

I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis.

I am using science, ie., abduction, to compare a class of hypotheses.

P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

is much much much lower than,

P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

[achrono]

This might be a naive question, but here me out. Do we really know what the difference is between statistics and the capacity to think? Is "true understanding" rather a continuum of sophistication from a simple adder to Albert Einstein?

My point here isn't "if it quacks like a duck...", but more so that while we are talking about intelligent apparatus we should be comparing apples to apples, and not say "this is a mere engine and that is a living brain".

[lgessler]

Idk, that isn't the sense I got from "It is absolutely trivial to show Hyp2 is false", but sure, I agree with you that this evidence certainly ought to tip the scales one way and not the other.