> doesn’t it then mean that whatever process is encoded in the NN, it should both be possible to represent in some more efficient representation...?
Not if NNs are complex systems[1] whose useful behavior is emergent[2] and therefore non-reductive[3]. In fact, my belief is that if NNs and therefore also LLMs aren't these things, they can never be the basis for true AI.[4]