[hgsgm]

The controversy over chatgpt standardized test performance is this:

To some extent, logic can cover lack of knowledge, and vice versa. Pattern matching mixes in too.

ChatGPT has incredible knowledge abd also pattern matching, and terible logic. (But a pretty good pseudo logic based on human language patterns, including human reasoning in written form.)

Chat got does well on tests using its incredible knowledge to cover it's lack of basic logical ability.

[garyrob]

I think you make a good point, except that I strongly suspect that when humans write software, etc. etc., they, too, are relying on patterns stored in their memories more than they are performing "fresh logic".

This is my impression, as someone who writes software professionally (staring in the 80's) and is now using ChatGPT as an assistant. I count myself in the group of people that don't use fresh logic all that often in coding. It's pretty rare that ChatGPT couldn't do the same things I do, and I see no reason to think I'm doing them in a more purely-logical way. At least not the vast majority of the time.

But I think you're making the point that humans at least have the ability to perform fresh logic, whereas ChatGPT may not. Maybe we differ in where the cutoff is that humans actually use that ability. I think it's pretty rare. I submit that it resides in times when people make the conscious decision to very consciously follow a series of very simple logical steps. That takes effort. It's not natural to us, although it may be more natural to some people than others. And I think that most people, most of the time, rely on pattern-based pseudo-logic instead of doing that.

[anktor]

Isn't the paper discussing precisely the opposite? That chatgpt predicts text in a way that, with each version, resembles more and more human logic, both with its succeseses and errors?