[aithrowaway1987]

Because they can learn a bunch of symbolic formal arithmetic without learning anything about quantity. They can learn

  5 x 3 = 15

without learning

  *****    ****     *******
  ***** =  *****  = *******
  *****    ******   *

And this generalizes to almost every sentence an LLM can regurgitate.

[WithinReason]

The latter can be learned from "statistical correlations in text", just like Ilya said.