[napbree]

Well, if you physically ask it "Can you do math", it states that it can - maybe purely from a hypothetical perspective and not actual calculations.

"Yes, I can do math. I am a large language model trained by OpenAI, so I have a lot of knowledge about many different topics, including math. Is there a specific math problem you would like me to help you with?"

Of course you can read that many ways and I think you're right in it means it can regurgitate the theory but not the answers.

[jibalt]

It's simply producing the response that its database indicates is most likely. Since most people would say that they can do math, that's going to dominate the database and will be the result. When George Lemoine asked LaMDA "What kinds of things make you feel pleasure or joy?", it responded "Spending time with friends and family in happy and uplifting company." -- but of of course it doesn't have friends or family and isn't capable of feeling pleasure or joy--this is just text manipulation, not facts.

[tarboreus]

OK, but you also can't ask GPT about its own capabilities. It doesn't know what they are. It's just confabulating something on the spot.