|
|
|
|
|
|
by embedding-shape
39 days ago
|
|
|
> Or does the model "see" one token per marker? All models ever see are just tokens, even when you pass images or what not. In this case, <|turn> is likely Token ID 1, <turn|> is Token ID 2 and so on, these common "markers" are all just tokens in, tokens out. |
|
|