[gavinray]

Can someone ELI5 how this hypothesis could ever be true?

  > "The core hypothesis: what makes language useful for pre-training is its structure, not its semantics."

As a layman, I've always held the intuition that semantics are the only meaningful thing.

"Structure without semantics" = form without function, symmetric/regular noise, right?

My naive bet is on compressing semantics into mediums more expressive/information dense than text. Like how some languages have single words/symbols to represent entire sentence-long concepts.

[andy12_]

I think what they mean by this is that, for example, in "If it's raining the outside is wet. It's raining, so the outside is wet", it's more important for the model to learn "If A then B. A, therefore B" than to learn what "raining" , "outside" and "wet" mean.