[cma]

> which themselves are not linguistic in nature (though of course the causality is so complex that the may be _influenced_ by language among other things).

Its possible something like this could be said of the middle transformer layers where it gets more and more abstract, and modern models are multimodal as well through various techniques.