[astrange]

LLMs also don't work by generating probability distributions of the next word. Your explanation isn't able to explain why they can generate words, let alone sentences.

[qsera]

That is exactly how they work.

[astrange]

No, a token is not a word.

[qsera]

I mean, it is some text.

[astrange]

How do you get from a piece of text smaller than a word to an entire coherent sentence?