[jjk166]

Distilling knowledge is data compression.

[w10-1]

You're conflating memorization with generalization, no?

[jjk166]

Memorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.

[staunton]

> Generalization is developing the heuristics by which you compress stored data

such that

> you nevertheless can recover enough information to be useful in the future.

I disagree (in case you meant to imply it) that compression implies generalization.

[jjk166]

I did not mean to imply compression implies generalization, if anything the reverse. Compression is the act of cutting, generalization is the whetstone by which you may sharpen a blade, which is the compression heuristic. A more general heuristic is to compression what a sharper blade is to cutting.