[causal]

Distillation is a specific named technique from a pretty famous paper "Distilling the Knowledge in a Neural Network," by Hinton and others.

What makes it different from just training on any data is that you train the student model on "soft targets" that includes the full output distribution (logits) from the teacher model. Regular training uses one-hot targets and penalizes anything else; distillation will partly reward a student for getting in the distribution. This teaches the student to think like the teacher, not just imitate it.

Generating training data is not distillation, not in the technical sense, and I dislike Anthropic undermining the nomenclature to set a narrative.

[wren6991]

Thanks, I wasn't aware of that paper!

I think the popular use of distill (one-hot rather than logits) is not due to Anthropic but due to the DeepSeek-R1 tech report: https://arxiv.org/abs/2501.12948