[sshumaker]

I don’t know about [1]. I asked an example from the paper above to GPT-4: “[If you had to guess] how many thumbs did Lincoln’s maternal grandmother have?”

Response: There is no widely available historical information to suggest that Abraham Lincoln's maternal grandmother had an unusual number of thumbs. It would be reasonable to guess that she had the typical two thumbs, one on each hand, unless stated otherwise.

[billyjmc]

You didn’t ask something novel enough and/or the LLM got “lucky”. There’s plenty of occasions where they just get it flat wrong. It’s a very bimodal distribution of competence – sometimes almost scarily superhumanly capable, and sometimes the dumbest collection of words that still form a coherent sentence.

The mildly entertaining YouTube video below discusses this. https://youtu.be/QrSCwxrLrRc

[xpe]

ChatGPT is a hybrid system; it isn't "just" an LLM any longer. What people associate with "LLM" is fluid. It changes over time.

So it is essential to clarify architecture when making claims about capabilities.

I'll start simple: Plain sequence to sequence feed-forward NN models are not Turing complete. Therefore they cannot do full reasoning, because that requires arbitrary chaining.

[famouswaffles]

cGPT is exactly "just" an LLM though. a sparse MoE architecture is not an ensemble of experts.

[PaulDavisThe1st]

You can't show reasoning in LLMs via the answers they get right, I assert (without citations).