[wilg]

It absolutely did understand it in some sense, as it completed it with a correctly structured sentence instead of random bytes.

I'm not sure we should expect it to know math, since it's not clear that, in a dataset of all text on the internet, the rules of math are not particularly important for it to learn. They're certainly in the dataset in various ways, but it has a limited memory and it is not surprising it knows other things much better.

We know math has hard rules because we learn that in other ways that this system does not implement. I think its at least remotely plausible that similar systems, with human-scale (i.e. orders of magnitude larger) datasets as input, taught in a similar way as humans, could demonstrate similar capabilities.