[yodon]

If you look at convolutional neural nets used in image processing, it's super common for the first layer or so to learn a family of wavelet basis functions. Later layers then do recognition in wavelet space, without that space ever being explained or communicated to the training algorithm.

This work here is obviously more complex than that, but suggests something similar is going on with early layers transforming to some sort of generalized basis functions defining a universal language representation.

[krackers]

Platonic representation hypothesis? https://arxiv.org/abs/2405.07987

Maybe in the same way fourier/wavelet basis is just the most natural way to work with certain signals or images, there's a certain representation for language and a representation for "thinking" that's natural. Maths could itself be said to be an abstract representation of the latter, and even if the idea of a "universal grammar" is dead, maybe there's still some natural space to work with all human languages owing to the shared biological priors we have as humans.