[jsrozner]

Interesting. I have been thinking that with these high dimensional representations that we have nearly infinite nearly orthogonal dimensions.

One thing that's interesting to me is where / how the model stores the info about a preference for a particular animal, and that this (presumably small) weights change leads to a difference in random numbers that then leaks into a student model.

The fact that this does not happen on models that are separately initialized/ trained could be seen to provide counter evidence to the recently published Platonic hypothesis paper.