[didibus]

> the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different

I'm not understanding, can you explain this more? How does it become more than a next token predictor? Isn't the post-training simply altering the sampled distribution? And isn't that distribution naturally occurring? It's the distribution of "useful" next token?

[IanCal]

IMO that doesn’t sound so much like prediction any more.

It’d be prediction if it’s “predict what would come next in this text sampled from distribution X”.

But what’s it predicting if we’re looking for new useful outputs? It’s finding a distribution that’s useful, and generating tokens, but it’s not predicting what comes next in a known sequence.

[didibus]

> predict what would come next in this text sampled from distribution X

It tries to learn the distribution of "useful" results either through verified rewards or human feedback. Then it encodes that in the network. When you run inference later, it samples or selects from that distribution.

Maybe it's a matter of interpretation. It's not predicting the next token based purely on the training corpus's distribution anymore, the RL process fine tunes that distribution so it predicts the next token that is closer to what was rewarded during RL. But as I see it, it's still predicting the next token, just from a reenforcement learned distribution instead of one found in a corpus of data.

[ruszki]

Just because you change loss function or dataset, it doesn’t become not a prediction. Just a prediction of something else.

[IanCal]

What’s it a prediction of?

[ruszki]

Prediction of the next token based on dataset and loss function.