[globnomulous]

This is essentially John Searle's Chinese Room Argument against strong AI. His conclusion is broader -- he argues categorically against the very possibility of so-called "strong" AI, viz. AI that understands, not just against the narrower notion that LLMs "understand" -- but the reasoning is essentially identical.

Here's the Stanford Encyclopedia of Philosophy's superb write up: https://plato.stanford.edu/entries/chinese-room/ It covers, engagingly and with playful wit, not just Searle's original argument but its evolution in his writing, other philosopher's responses/criticisms, and Searle's counter-responses.