|
|
|
|
|
|
by wren6991
3 days ago
|
|
|
> Proper distillation requires access to the logits Why do you need logits? Can't you just train on cross-entropy loss of the model against the hard decision, like you do in regular pretraining? There are definitely current-gen open-weight models (Step 3.7 Flash is one) that refer to themselves as an OpenAI model in CoT, but not in the final response. |
|
|