[Erndob]

Besides how well something works, I am really curious if there’s any divergence that comes from different grammar in languages.

As in, the way languages are structured is different. Some are more precise, some are less, the information density per syllable is different, etc.

So besides just pure performance due to differences in training data, I’m curious if there’s some fundamental difference in the way LLMs interact with data in different languages even if end information is the same. Because even just in English, phrasing slightly different can yield different results.

Edit: would be interesting to see the “thinking” of the model done in different languages. Is identical problem thought about more or less the same, or does agent go on different train of thought depending on the language it is thinking in?

[curioussquirrel]

I am fairly convinced that there's a certain polyglot snowball effect: once the LLM is fluent in 20 languages, it can pick up on similarities in vocabulary, syntax etc. and learn the 21st language with much less effort (and training data). This might be difficult to actually study in an isolated way, but it's a real effect for humans and it makes sense the the pattern matchers that LLMs are would find these shortcuts.

Using similar words should land you in similar places in the latent space, even if they actual word or their order is slightly different. Where it gets interesting is how well English words map to their counterparts in other languages, and what practical differences it makes. From various studies, it seems that the gravitational pull of English language/culture training data is substantial, but an LLM can switch cultures and values when prompted in different languages.

[curioussquirrel]

Just saw your thinking edit! That's a great question and one I wanted to study in depth, but these days you don't really get access to the raw thinking data. It's usually summarized and you can't even be sure what language the model thought in unless you have access to the logits (so only viable for open-weights models).