Excellent point. See also the failure of Sapir-Whorf to prove that language determines thought. I think we have plenty of evidence that, while language can influence thought, it is not thought itself. Many people invested in AI are happy to throw out decades of linguistic evidence that language and thought are separate.
I was with you up to the last sentence. By what reasoning do you claim that LLMs only consist of words? The input and output are words but all the stuff in the middle - where the magic happens - does not appear to be quite that simple.