[zmgsabst]

“A is B” doesn’t generally entail “B is A”.

“A square is a rectangle” does not entail “a rectangle is a square”.

Similarly, “Socrates is alive” doesn’t entail “alive is Socrates”.

Notably, they mention when context is included, LLM performance rises — ie, exactly when we include extra information that allows it to recognize what kind of information is being conveyed.

But the LLM is correct not to generalize that pattern when it doesn’t generalize — even if researchers have salient example, but ignore contrary ones (eg, square-rectangle or Socrates-alive).

[Dibby053]

"A is the B" does entail "the B is A", because "the" establishes an identity/bijection.

[WithinReason]

That completely misses the point. The point is that "Valentina Tereshkova was the first woman to travel to space" does imply "The first woman to travel to space was Valentina Tereshkova", which LLMs fail to recognise.

[IshKebab]

Right, but that could be because the fact that that implication exists is not actually as trivial as they are implying.

Does "Flargbler was blorglargh" imply "blorglargh was Flargbler"? Maybe. You need more context to know.

[1718627440]

> You need more context to know.

Yes, which is what the model is. That's the point here, it doesn't has or accesses this context, even though it should.

[IshKebab]

It seems to me that the point was usually "these models are fundamentally not intelligent because they have this incredibly dumb failure modes" rather than "these models happen to have this interesting weakness".

Maybe the original authors didn't make that mistake, but that was certainly the case whenever it was brought up by HN commenters of the "stochastic parrot" persuasion.