[l33tman]

The internal model will certainly pick up on statistical correlations among the text analysis corresponding to an 8x8 2D grid as this is the most low-hanging statistical representation that helps solving the problem during training.

The same argument and result exist for the different human sensory modalities - neurons and connections self-organize to have the same topology and layout as the retina (2D) and frequency / time for the audio (also 2D).

In fact, wasn't this experiment already done for Othello and LLMs recently? Wasn't there a paper where they found the internal model for the board?

[IIAOPSW]

It can learn the rules for movement strictly as generator rules imposed on a string of tokens representing the previous sequence of moves. Each new item appended to the list has to in some way match a previous item in the list. Eg RC6 is a Rook, so it has to match an earlier token that is also a Rook, in one of two ways: R_6 or RC_ (and it must not be previously captured by __6 or _C_ ). At no point is it even necessary to convert the move history into a present board state, let alone the state of an 8x8 grid. The move history is sufficient board state on its own. Are the rules for valid chess moves, expressed as 3 character token grammar, the same thing as having learned a 2d grid in latent space? I don't think so, because the language rule is more general and isn't restricted by geometry.

In principle it could reason about any incidence structure. That is, anything where the semantics is two types of objects, and a "touching" relation between them. Lines are just all the points along them, points are just all the lines intersecting there. For the purpose of directions, a train station is just a list of all the services that go there, and a service is just the list of stations where it stops. Etc etc. A language model is free to learn and understand these sorts of systems purely as relations on symbols without ever implicitly organizing it into a geometrical representation.

This is all good news. It means Chess, Transit, Diplomacy, and many other things can fit nicely into pure language reasoning without trying to ground the language in the semantics of our physical nature with its dimensions of space and time and whatever.

What would change my mind is if, after learning the rules for Chess as string matching, it invented a word for "row" and "column" on its own.

[yorwba]

That paper is at the link containing "othello" upstream.