[lern_too_spel]

Transformers have shown that language development can also be explained by what you derisively call reflex behavior, with no need for specialized Chomskyan recursion circuitry, as the same mechanism with slight modification (decision transformers) also plays Pong.

[rryan]

There's a huge difference between fitting a probabilistic model to a data distribution then sampling from it (what GPT-3 is) and agents that invent language and use it to communicate.

[lern_too_spel]

Not much. A transformer trained on multiple senses can learn the sound that an animal makes and associate it with seeing that animal. It can also learn how another agent reacts after it says a word.

The huge difference is actually between animal reflexes and learned behavior. Reflex is built-in. I didn't learn to kick my leg in response to a tap on the patellar tendon.

[rryan]

I agree that a Transformer is an example of a "reflexive" behavior because it learns to react in a context (via gradient descent rather than evolution as the learning algorithm). It's a conditional categorical distribution on steroids.

I also agree it's not much different than what's going on in this petri dish with pong.

But I don't think that's a profound statement.

What I'm saying is that calling what a Transformer does "language development" isn't accurate. A Transformer can't "develop" language in that sense, it can only learn "reflexive" behavior from the data distribution it's trained on (it could never have produced that data distribution itself without the data existing in the first place).

[lern_too_spel]

> I agree that a Transformer is an example of a "reflexive"

I said that it is not reflexive. It is learned. Just because after you learn something, it becomes easy does not mean that it is a reflex. I explained why language development can be done with little more than a transformer learning from how others behave when you make an utterance and from how you behave when you hear something, like a decision transformer learning what happens after it takes certain actions in Pong.