Hey! I'm working on exactly this same problem for an undergraduate Computer Vision course right now. It's not going well! Chess piece recognition is hard. I definitely think the electronics approach described in the other comment is a much more reliable way to go.
If you ever do get it working, I think it'll be fun to push it to over-recognize chess games everywhere, like the deepdream stuff over-recognizing animals (or whatever) everywhere.
This might be hugely easier if you can keep state between turn analyses. IE, if a white rook disappears from A5 and some unidentified white piece appears at C5, then it’s probably the rook.
We weren't given many constraints related to the chess piece recognition itself. The course instead asks us to implement a CV research paper, and we chose an existing research project which focused on chess piece recognition.
That lack of constraints led us into running face first into issues of generalisation and variability within datasets. As in, exactly what you allude to with limiting the piece sets.
I think in my undergraduate naivety my aspirations were too high with what could reasonably be accomplished. I've spent a lot of time trying to improve an aspect of the project that really didn't need to be improved, which prevented meaningful progress.
Now finals are coming up and I feel terribly stressed. Having trouble functioning. Brain fog, etc. I feel so sad right now.
EDIT: I keep forgetting my password so apparently I have multiple throwaway now. Sorry.
It takes a lot of shortcuts, works with just the right lightning, etc., but worked great as a proof-of-concept :)
We got away with not identifying the pieces by just detecting the color, assuming the game started from the initial position, and assuming only legal moves (the whole game is unambiguous using these assumptions).
It's all old-school computer vision with hand-written features, and I'm pretty confident there is tons of low-hanging fruit, but who has the time.
If I understand correctly (watched the video) you start from the initial position and track moves that are made in order to update the position to reflect the move, is this correct?
That's correct. If you're curious about the architecture: the vision part detects a chessboard, then corrects the perspective and restricts the image to just the square of the board, then looks at each square and has some simple thresholds to decide if it's occupied by a piece, and of what color. From there, the camera is treated as a black box sensor that continuously streams two 64-bit masks, for where it thinks it sees white and black pieces. There is a second program (controller) that turns that stream into a stream of chess positions (and a Unix pipe in between). The sensor is faulty of course and the controller has logic for ignoring bits from the mask where there cannot possibly be pieces etc.
Those boards are ~$1000 each. High level tournaments use them, but low level tournaments don't unless some rich person has bought the club a bunch. Most tournaments are low level and are designed not to turn a profit: the entry fees either pay for the room rent, or go to the prize winnings: the more people who enter the more prize money there is.
Yeah, the DGT board is perfect for on-the-fly tournament analysis -- but a computer vision tool would be great for times when you don't have access to the/a DGT board or its data
Set one up at the park and get some insight on those hustled blitz games!