[Jensson]

Humans add structure to unstructured data, the AI models we have just make flawed replications of structure we feed it, that is the difference. ChatGPT didn't even figure out the structure of basic math which is the most salient logical structure humans have. Even idiot humans can learn to count and compare quantities without being hardcoded to do so, they learn it from just words and pictures. A language model that failed to learn this when it was trained on the entire internet therefore can't have any understanding the way humans understands things, and feeding it more compute or data wont get it there either.

Example of a question a typical human idiot can solve without ever getting it wrong, but ChatGPT can't reliably: Is 7 dollars enough to buy a thing costing 7 dollars? ChatGPT can usually solve this, but it sometimes gets it wrong, getting that sort of thing wrong ever means that it doesn't understand, it just uses dumb statistics.

Edit: I'm not saying it is impossible to make AI models that understand these things, just that the ones we have today don't.

[chatgptexpert]

Dumb statistics you say?

https://i.imgur.com/im9EquK.png

Perhaps it's magic trick that a language model can do math, but it's quite impressive that it gets very close to the desired answer.

[Jensson]

It probably had problems with this part "two one dollar bills". A lot of text that contains "two" and "one" results in a "three", but also many texts with "two" and "one" results in "two", and then the model randomly chooses between those two interpretations. The worst part is that it doesn't even do it consistently for the same piece of text, creating that nonsense.