[pixl97]

The token model of LLMs doesn't map well into how human experience the world of informational glyphs. Left and right is a intrinsic quality of our vision system. An LLM has to map the idea of left and right into symbols via text and line breaks.

I do think it will be interesting as visual input and internal graphical output is integrated with text based LLMs as that should help correct their internal experience to be based closer to what we as humans experience.

[lawlessone]

" An LLM has to map the idea of left and right into symbols via text and line breaks."

Oh yeah that's i suggested it :)

I do wonder though if we give the LLMs enough examples of texts with people describing their relative spatial position to each other and things will it eventually "learn" to work things these out a bit better

[famouswaffles]

>I do wonder though if we give the LLMs enough examples of texts with people describing their relative spatial position to each other and things will it eventually "learn" to work things these out a bit better

GPT-4's spatial position understanding is actually really good all things considered. By the end, 4 was able to construct an accurate maze just from feedback about the current position and possible next moves after each move by GPT-4.

https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-...

I think we just don't write much about moving through space and that is why reasoning about it is more limited.

[astrange]

A funny thing GPT-4 is unusually good at is giving driving directions. This shouldn't work, and of course isn't 100% right, but… it's kind of right.

Bard can answer questions like this, but I think it actually uses the Maps API. (It certainly says that's what it's doing.)

On the other hand, every chatbot including GPT-4 is both unable to do ASCII art and unable to tell it can't do it. (Bard always shows you `cowsay` and tells you it's what you asked for, no matter what it was supposed to be.)

[bt1a]

I tried so hard to make ascii art with GPT-4 api :(

[walnutclosefarm]

Probably. But what seems much more interesting is to have a spatial model pre-seeded in the LLM, so that it "attaches" language to that as part of its training. Ditto for other models of the world we want the language module to be able to draw on and reason with.