[daveguy]

> It is showing the same test applied to a human.

But you misrepresented the test with respect to humans. Humans who know how to play chess don't make illegal moves.

> That said, humans who have written entire books on chess have been known to play illegal moves.

Citation needed. Unless you are talking about stories from when they first learned the rules?

[Lerc]

https://www.chess.com/blog/kranthimanaswi/top-5-illegal-move...

[daveguy]

Did you read those? These are the "illegal" moves listed:

5. Mouse slip

4. Forgot to call check

3. Accidentally touched 2 pieces, tried to fix it

2. Forgot to hit the clock button

1. Castle through attacked square

So, the only one of these that was an acual "illegal move" of the sort LLMs make was the castle through attacked square.

LLMs sometimes just move pieces wherever. And that does not happen when humans who know the rules play. Yes, they may mess up en passant or promotion too. But a basic "how a single piece moves" rule is what LLMs f up.

[Lerc]

I wouldn't count mouseslips as legitimately illegal moves either, they are also incredibly rare because most online players play with auto confinement to legal moves.

Moving through check definitely counts as as an example of a human knowing the rule and yet playing the move anyway. Which was the position you took when claiming humans would not do moves against rules they have learned.

In my experience sub 2000 players playing OTB informal chess do illegal moves fairly regularly, perhaps 1 in 50 games. Moving knights one square too far, slipping a bishop from one line to the next on a long diagonal. Castling after moving the king, not moving out of check, moving into check (especially by moving a pinned piece)

They all meet the criteria of knowing the rules and playing something else. Oftentimes people do this because they have a mistaken assumption about board state. I suspect the same is true for LLMs, they are making valid moves for what they mistakenly think the board is. That would be difficult to test, but I think possible with the right introspection tools.

[daveguy]

Not sure how you don't see the difference between an LLM f'ing up how a single piece moves vs forgetting to hit the clock, accidentally touching two pieces or forgetting to call check. At least we agree and recognize that a mouse slip as different. Seems like some serious apologizing/rationalizing for LLMs on the other "moves". Anyway, have a good day, buddy.

[Lerc]

Well I only addressed the mouse slip because that was the one you hilighted becore you edited you post to include the others.

I doubt any of it was rationalising for LLMs considering I was trying to address the contention that humans do not make moves counter to rules that they know. The performance of LLMs has no bearing on that claim one way or another.

[daveguy]

So you hadn't read your reference before you read my post? If so, you would have known the only illegal chess move was a missed attack square between a castle. For the record I didn't see any of your response before I completed it. Didn't realize you were going to jump to defend so quickly.

Well, I hope your day is going well. Keep on cheerleading.