[ganz]

The system isn't training on antonyms and analogies - it's training on wikipedia. It's learning the meaning (and multiple senses) for every word it can find.

The test they use to see if it actually learned what these words meant, in a limited sense, is to test it against a subset of verbal IQ tests (not what it was trained on!). You could ask it the antonym, synonym, or analogy for anything in English. This is an extension of word2vec / word embeddings.

That it beats the scores of college graduates impresses me.

[fauigerzigerk]

"it's training on wikipedia. It's learning the meaning (and multiple senses) for every word it can find."

I don't think that is entirely correct. After cursory reading of the paper, my understanding is that they look up a list of word senses for each word in a dictionary (or multiple dictionaries). And then they try to learn something about each of those word senses from wikipedia (that is they create seperate word embeddings for each of those senses). So what they do not do is to learn what senses a word has. That is done by the humans who created the dictionaries.

What that means is that they cannot pick up new senses of words, which doesn't matter for answering IQ test questions because these questions rarely change and are typically based on well established word meanings.

Unfortunately it makes this approach less than ideal for things like understanding the news (something I'm working on), where new contexts of words keep popping up all the time.