[sigbottle]

You could redefine language to include mental abstractions and maybe it wouldn't be so weird.

Language is a shared medium. Maybe an intelligence is capable of building abstractions in their head without the need for say, a symbol system to hopefully communicate with species similar to itself.

You may argue that the agent needs persistent memory. Well, so what? If the world has stable regularities from the perspective of the agent - if the sun rises every day, and language is associating certain visuals with certain thoughts (it's not when viewed in its original social definition - but conduct the thought experiment with sociality removed), who's to say that you can't form a kind of language around that?

Of course, this is all armchair, and I'm not trying to separate out what "real language" is or whatever, just thought experiments.

But the original poster has a good point though, that's been proven in papers, that the tokens an LLM spits out, themselves, attract them to certain distributions, and at least a decent amount of the tokens in CoT are there solely to bring itself to the right distribution.