[abm53]

You can ask ChatGPT to solve maths problems which are not in its training data, and it will answer an astonishing amount of them correctly.

The fact that we have trained it on examples of human-produced maths texts (rather than through interacting with the world over several millennia) seems like more of an implementation detail and not piece of evidence about whether it has “understood” or not.

[dbsmith83]

They also get problems wrong, in the most dumb way possible. I've tested it out many times where the LLM got most of the more 'difficult' part of the problem right, but then forgot to do something simple in the final answer--and not like a simple error a human would make. It's incredibly boneheaded, like forgetting to apply the coefficient it solved for and just returning the initial problem value. Sometimes for coding snippets, it says one thing, and then produces code which does not even incorporate the thing it was talking about. It is clear that there is no actual conceptual understanding going on. I predict the next big breakthroughs in physics will not be made by LLMs--even if they have the advantage of being able to read every single paper ever published, because they cannot think.