[dash2]

Arithmetic seems like an example where it would help to learn from the real world, not just from text. I learnt to add up by watching my teacher manipulate plastic Lego-style blocks. Put 3 blocks with 2 blocks, and you have 1, 2, 3, 4, 5.

[londons_explore]

But somewhere in that massive corpus of text will be a description just like you've just given...

[tsimionescu]

Sure, but GPT-3 doesn't attach semantics to text, it just learns how to produce text patterns that are similar to text patterns it has seen before.

[FeepingCreature]

"Similar" is an inherently semantic property.

[tsimionescu]

Would you say that something like string.GetSimilarity(string), which tells you by what number of characters two strings differ, is interpreting the text?

Basically what GPT-3 does is to find a string X of a particular length such that it maximizes concat(userInput, X).GetSimilarity(someStringInTrainingSet).

Edit: to be clear, I'm not suggesting it's looking up the training set at runtime, X.GetSimilarity(someStringInTrainingSet) is basically what got baked in during training.

[FeepingCreature]

That is not what GPT does.

[JonChesterfield]

Words with the same length are similar to one another but not well correlated semantically. E.g. bog vs dog.