Why should AI enthusiasts expect logical reasoning?
AI "reasoning" is the model babbling to itself, using statistically likely sequences of words that appear in the training data. Of course that will often produce valid chains of logic, but not because it in any way understands the principle behind it. It is ONLY modelling LANGUAGE, not any of the mental processes that guide the use of language in humans (and which might even exist in animals not capable of language).
When asked to carefully check itself, it might spot errors because their structure matches common examples of fallacies, or it might not, and instead insert some further bullshit like "Assumption Verified — 100% correct! It all checks out [insert some emojis here]". That then becomes part of its context, and further boosts its unfounded confidence.
AI "reasoning" is the model babbling to itself, using statistically likely sequences of words that appear in the training data. Of course that will often produce valid chains of logic, but not because it in any way understands the principle behind it. It is ONLY modelling LANGUAGE, not any of the mental processes that guide the use of language in humans (and which might even exist in animals not capable of language).
When asked to carefully check itself, it might spot errors because their structure matches common examples of fallacies, or it might not, and instead insert some further bullshit like "Assumption Verified — 100% correct! It all checks out [insert some emojis here]". That then becomes part of its context, and further boosts its unfounded confidence.