[evanhu_]

I'm super impressed with what you've managed to create, do you have any further plans with this project? I'm curious now that it's finished and documented to such an extent will you try to bring it publicity and actual usage or was this just a passion project. Thanks

[cbutner]

Thank you! I do get that itch to jump in and improve things whenever I see it lose a game, but I don't have further plans (development or commercial) in the near-term. The goal originally was to see whether I liked ML, to decide on my next industry/career move, but there was a lot of "one more month".

I'm actually hopeful that some search techniques such as SBLE-PUCT[1] or better derivations can make their way into other open source projects, but they've had big teams working for a while on similar, often better ideas, so we'll have to see.

[1] https://chrisbutner.github.io/ChessCoach/high-level-explanat...

[mxwsn]

So, do you like ML?

[cbutner]

Haha - I dislike how much of a black box it is, despite the statistical basis (for example, the back and forth on batch normalization rationale). But lots of interesting problems and tech to dig into.

[ritchiea]

You estimate it’s rated 3400-ish and it loses games????

[cbutner]

It loses some games to Stockfish 13 and 14, and Lc0 - rarely at slow time control, and more often at blitz and bullet (actually, it has losses all the way down to Stockfish 9 in blitz).

Partly because of the way it tries to search more widely to avoid tactical traps, it can also be a little sloppy in holding advantages or minimizing losses (this could use some more work and tuning). This ends up making it a little drawish, so it loses less than you'd expect to Stockfish 14, but also doesn't beat up weaker engines as well as Stockfish 14 does.

You can see some of this in the raw tournament results[1]. At 40 moves per 15 minutes, repeating, each engine draws with the ones above and below it, but starts to win and lose at a distance of 2 or 3.

At 5+3 time control, ChessCoach goes 1-0-29 vs. Stockfish 12, but Stockfish 12 is better at beating Stockfish 8-11 than ChessCoach is, so CC ends up between SF11 and SF12 in the end.

On Lichess, where there's no "free time" to get ready for searches, ChessCoach's naïve node allocation/deallocation makes it waste time, and means it can't ponder for very long on the opponent's time - a big opportunity for improvement (it needs a multi-threaded pool deallocator that can feed nodes back to local pools for the long-lived search threads). I think it's also hitting a bug with Syzygy memory mapping that Stockfish works around via reloading every "ucinewgame" (which I don't trigger on Lichess). So, overall, its performance on Lichess is worse.

Also, you can't read too much into this data - very few games, and no opening book.

[1] https://chrisbutner.github.io/ChessCoach/data.html#appendix-...