I suspect the truth (if this claim is true) is a lot more nuanced than "it did grade-school math", and there's more context surrounding this claim which insiders have and makes it much more interesting.
Like most research they likely started with a smaller model like GPT 2 or 3 and shown that they can significantly boost the performance to the extent of solving grade school math.