[famouswaffles]

>It is absolutely trivial to show Hyp2 is false

No it's not

> Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain.

>Indeed: because there're no relevant prior cases to sample from in that case.

That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolutionary biology (some still look absurd) and that don't speak to any lack or presence of intelligence or complex thought. Not sure why it's so hard to grasp that LLMs are bound to have odd failure modes regardless of the above.

and trivial here is relative. In my experience, "trivial" often turns out to be trivial in the way a person may not pay close attention to and be similarly tricked.

For instance, GPT-4 might solve a classic puzzle correctly then fail the same puzzle subtlety changed. I've found more often than not, simply changing names of variables in the puzzle to something completely different can get it to solve the changed puzzle. It takes memory shortcuts but can be pulled out of that. LLMs have failure modes that look like human failure modes too.

[mjburgess]

The "failure modes" in humans do not show we lack the capacity.

Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object.

> I've found more often than not, simply changing names of variables

Yes, lol --- why do you think that is?

Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case.

P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference)

To such a degree that the latter is vanishingly close to zero.

[famouswaffles]

>The "failure modes" in humans do not show we lack the capacity.

Then they don't in LLMs too

>Yes, lol --- why do you think that is?

Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.

[mjburgess]

> Then they don't in LLMs too

LLMs don't get drunk .

If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not.

They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.).

Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain.

But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics.

In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding.

Cognitive impairment = reasoning machine is broken

Environmental illusion = data is ambigious and actions cannto resolve it

These "failure modes" are expected if you actually have the relevant capacity.

[moffkalast]

> LLMs don't get drunk .

Well actually they sort of can...

https://www.reddit.com/r/LocalLLaMA/comments/13vv941/tempera...

[famouswaffles]

>Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain.

alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?

[mjburgess]

see, just one reference in the paper: https://arxiv.org/pdf/2302.08399.pdf

[famouswaffles]

they can answer those

https://medium.com/@nathanbos/prompting-better-theory-of-min...

[YeGoblynQueenne]

>> alright let me humor you for a bit.

That's real class, right there.

[lostmsu]

> less so. But not if I permute the name of the object.

You need to realize that you wrote it on a forum where the most known joke is "there are two hard things in programming". That would immediately show you how this assumption is exactly false.

[lgessler]

This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define that.

[mjburgess]

I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis.

I am using science, ie., abduction, to compare a class of hypotheses.

P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

is much much much lower than,

P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

[achrono]

This might be a naive question, but here me out. Do we really know what the difference is between statistics and the capacity to think? Is "true understanding" rather a continuum of sophistication from a simple adder to Albert Einstein?

My point here isn't "if it quacks like a duck...", but more so that while we are talking about intelligent apparatus we should be comparing apples to apples, and not say "this is a mere engine and that is a living brain".

[lgessler]

Idk, that isn't the sense I got from "It is absolutely trivial to show Hyp2 is false", but sure, I agree with you that this evidence certainly ought to tip the scales one way and not the other.

[sgt101]

was with you until:

> look absurd outside the context of evolutionary biology

for humans, everything (everthing) is within the context of evolutionary biology!

> LLMs have failure modes that look like human failure modes too.

Yes - because LLM's are trained on 2020 Reddit.

[famouswaffles]

>for humans, everything (everthing) is within the context of evolutionary biology!

Sure but if some alien species were observing us, some of our actions would look downright odd. Evolutionary biology doesn't necessarily hold the same reference frame for other species, even on earth. Octopi are weird to us. Not so much to other Octopi.

>Yes - because LLM's are trained on 2020 Reddit.

I wasn't making any comment on why this was the case. Simply that it was. There'll be failure models LLMs adopt from training data, but there's also bound to be failure modes LLMs adopt from the training scheme itself.