[palmer_fox]

I was wondering about that too. Would it be possible in the future to have a more modular approach to LLMs? Have a module that is responsible for basic knowledge/language/grammar and then other more specialized modules that are added selectively.

I don't know enough about fine-tuning, not sure if the process is capable of removing "unused" parts of the model (I guess not possible, similar to un-learning).

[lucubratory]

There are various methods for removing unused parts of the model, like distillation. The idea is generally that you always lose performance, but hopefully you lose more size/runcost than you do performance, proportionately.