Hacker News new | ask | show | jobs
by HarHarVeryFunny 36 days ago
If you use "distillation" to generate some high quality training data from another model, then this in of itself says nothing about the quality of any model that you train using this additional data - it trivially also depends on how good that model would have been without that additional data, and also what the data is (maybe just for a narrow skill where more data can be very impactful).

This is different from what we normally think of distillation where a large model is distilled (with full access to logits, thinking traces, etc) into a smaller one such as opus -> sonnet, where the smaller model is 100% using data generated by the parent and may (without further training) be expected to be inferior to it.