[inigyou]

It means the word "the" as part of instructions and the word "the" as part of data would be two different tokens

[danlitt]

But tokens are just text! Isn't it all just text? If you're training and you encounter "the", is that an instruction "the" or a data "the"?