[WithinReason]

That completely misses the point. The point is that "Valentina Tereshkova was the first woman to travel to space" does imply "The first woman to travel to space was Valentina Tereshkova", which LLMs fail to recognise.

[IshKebab]

Right, but that could be because the fact that that implication exists is not actually as trivial as they are implying.

Does "Flargbler was blorglargh" imply "blorglargh was Flargbler"? Maybe. You need more context to know.

[1718627440]

> You need more context to know.

Yes, which is what the model is. That's the point here, it doesn't has or accesses this context, even though it should.

[IshKebab]

It seems to me that the point was usually "these models are fundamentally not intelligent because they have this incredibly dumb failure modes" rather than "these models happen to have this interesting weakness".

Maybe the original authors didn't make that mistake, but that was certainly the case whenever it was brought up by HN commenters of the "stochastic parrot" persuasion.