[adrianratnapala]

I suspect another way of looking at it is that these models actually learn about the real world by reading about it in the WSJ -- of course their knowledge of it is not as deep as our own, but good enough for what they do.

That is, if you took the well NLP trained model, then you could in principle extract out of it facts like "streets are not found inside cars".