[callmekit]

So yesterday we had a post (https://news.ycombinator.com/item?id=35200267) that you need a special "play as a grandmaster" prompt to reduce the number of illegal moves for GPT3.5, and that "GPT4 sucks" at chess completely compared to GPT3.5.

Now we have this post that GPT4 plays good and doesn't make illegal moves at all. What changed? What was the prompt? Is it just random noise?

[ctchocula]

If you've checked the author's previous post [1], you'll see that he admits to being 1200 Elo on chess.com, which is beginner / early-intermediate level. So him losing to GPT-4 may not mean much. Maybe that explains the supposed contradiction here.

[1] https://villekuosmanen.medium.com/i-played-chess-against-cha...

[owenmarshall]

The problem with ChatGPT seems to be that it often gives answers that appear plausible at the surface, but with enough knowledge you realize they are inaccurate or even wrong. I wouldn’t put much stock in the analysis of a beginner - I’d trust them to say the moves were legal, and that ChatGPT stopped trying to materialize pieces from thin air, but not any analysis beyond that.

[ChatGTP]

Disclaimer: YMMV