Can't you take a broadly trained model and throw away useless knowledge (like foreign languages, and 99% of wikipedia) and still end up with a model that is as intelligent and useful as the original but much smaller?
Models are not a collection of training data. The entire model is shaped by the training data it sees during training.
What we call knowledge is encoded as weights. Something we identify as knowledge is the sum of many weights across the model. However the same weight may be involved in very different topics. You can’t point to every weight and say that this one is only for a part of Wikipedia or Harry Potter that I don’t need. That weight could be active for a multitude of topics.
There have been a lot of attempts to reduce the size of mixture of experts models by selectively removing experts. The results are not good, but knowledge isn’t neatly separated within the model.
English reasoning is far better when the model has been trained on many other languages even if you only work in English and never touch another language.
I suspect it's a bit like tree-shaking/dead-code elimination in an untyped, dynamic language. You can probably delete a lot of things, but each deletion has a chance to break (or silently degrade) some functionality. Good coding practices, good docs and comments, and good tests can help determine whether a given definition is safe to delete or quickly verify that the deletion was a mistake after the fact. LLMs' weights lack all of those. It's like trying to perform DCE on a binary linked from outputs of ten different compilers with different optimization levels (and the linker is outright bonkers) without running the binary once. Oh, and the binary is many tens of GB in size. Yikes, I'm getting goosebumps just imagining it!
I'm not an AI researcher (would love to hear an actually informed answer on this!), but AFAIK the weights are more or less a black box - you can't easily pinpoint that weights x, y and z relate to Ancient Rome and can therefore be safely removed from a model that's optimised for coding.
if you have the model weights or a provider that will give you the full token distributions you can have the next best thing and do a domain-specific distillation. this way you can take a 9B or 27B LLM and uplift it to the level of the larger model.
it still takes some work and ingenuity, you need to design a workflow that would let you efficiently probe the larger model with your domain-specific prompts, this usually involves a workflow where you obtain the inverse mapping from frontier models (don't need distributions for this step).
What we call knowledge is encoded as weights. Something we identify as knowledge is the sum of many weights across the model. However the same weight may be involved in very different topics. You can’t point to every weight and say that this one is only for a part of Wikipedia or Harry Potter that I don’t need. That weight could be active for a multitude of topics.
There have been a lot of attempts to reduce the size of mixture of experts models by selectively removing experts. The results are not good, but knowledge isn’t neatly separated within the model.