[klibertp]

I suspect it's a bit like tree-shaking/dead-code elimination in an untyped, dynamic language. You can probably delete a lot of things, but each deletion has a chance to break (or silently degrade) some functionality. Good coding practices, good docs and comments, and good tests can help determine whether a given definition is safe to delete or quickly verify that the deletion was a mistake after the fact. LLMs' weights lack all of those. It's like trying to perform DCE on a binary linked from outputs of ten different compilers with different optimization levels (and the linker is outright bonkers) without running the binary once. Oh, and the binary is many tens of GB in size. Yikes, I'm getting goosebumps just imagining it!