Right now we don't know if LLM's are also doing this, or not.
In their calculating the 'next' word, as part of that 'weighting', are the 'simple rules' for future words, that you are saying humans do but LLM's can't.
It doesn't seem LLMs are capable of "understanding" that they don't "know" something, so there are certainly some observable differences. But the intent of my original comment was precisely to highlight that we don't know. It could be that next token predication is somehow mathematically equivalent to what we call consciousness. That would certainly be a revelation.