[Spacecosmonaut]

I wonder, how does distillation deal with unprobed spaces in the knowledge landscape? Is a distilled model worse in some niche area that was not probed? Presumably, this is why frontier labs dont distill their own models internally to release them to the public as a servicable frontier model.

[tngranados]

Do frontier labs not distill their own bigger models into the smaller/cheaper variants? I thought that’s been the case for a while