The point is that it is providing an unexpected and novel behavior in spite of having zero programming to produce this kind of output. It's odd and almost implies a kind of self-awareness to "understand" the rules to a complicated game, and then use logic to predict what is an optimal move. Even though it's only defined input being an incredible amount of text from across the internet.
Is it going to be a good chess player? Not really. But likewise, a dog will never be a great basketball player; but if one manages to somehow get onto a team and demonstrate competency with the fundamentals and team play then I will be impressed too.
The goal isn't to use this to replace existing chess AI's, but to show how just a language model is able to perform a non-language related task very well. Also not only does it play correct moves (at a lower skill level than a proper chess AI), it is able to explain and justify it's move choice.
Is it going to be a good chess player? Not really. But likewise, a dog will never be a great basketball player; but if one manages to somehow get onto a team and demonstrate competency with the fundamentals and team play then I will be impressed too.