[WJW]

Even 1300s sometimes make the best move. Sometimes the best move is really easy to see or even mandatory, like if you are in check and MUST take that checking piece. Sometimes the best move is only obvious if you can look 20 moves ahead. Sometimes the best move is only obvious if you can look 5 moves ahead, but the line is so forcing that even 1300s can look that far ahead.

Despite decades of research, nobody has found a good way to make computers play like humans.

[throw310822]

Then I can't refrain from asking: and what's the style of LLMs? For example the ChatGPT which is apparently rated around 1800? That should be completely different from that of a classic chess engine.

[WJW]

LLMs can be trained on chess games, but the tree of possible board states branches so fast that for any given position there is simply very little training data available. Even the billions of games played on chess.com and lichess are only a drop in the bucket compared to how many possible board states there are. This would have to be split further by rating range, so the amount of games for any given rating range would be even lower.

This means that the LLM does not actually have a lot of training data available to learn how a 1300 would play, and subsequently does a poor job at imitating it. There is a bunch of papers available online if you want more info.

[throw310822]

LLMs already do play at elo ~1400-1800. The question was how does their style feels like to someone who can appreciate the difference between a human player and a chess engine (and the different styles of different human players).

[jquery]

I can’t speak for ChatGPT, but your intuition is correct that LLMs tend to play more like “humans” than Stockfish or other semi-brute force approaches.

[ludston]

ChatGPT will hallucinate and make impossible/invalid moves frequently, so I don't see how it could have a chess rating

[throw310822]

That's not the case. Depending on the version, (Chat)GPT seems to be able to play between ~1400 and ~1800 elo, very rarely making invalid moves.