[thingification]

Speaking from ignorance (I've not studied attention nor transformers): This is my feeling too. I feel like the next step isn't far away: a more explicit model of the world with a mechanism to both query and "feed back on" that model, correcting mistakes.

If it's possible to get so far when that functionality seems in an important sense basically missing, imagine how far it'll go when that does happen.