[drw]

Something we work very hard on at chess.com is the rating of the individual tactics puzzles.

Some of you might enjoy looking at the distribution of our puzzle ratings. Truly hard problems are precious, and we would always like to have more at the high end. The distribution is multimodal because puzzles of different lengths have different modes.

  +----------------+---------+
  | ratings_band   | count   |
  |----------------+---------|
  |   100-199      | 5320    |
  |   200-299      | 2542    |
  |   300-399      | 31542   |
  |   400-499      | 10282   |
  |   500-599      | 4539    |
  |   600-699      | 7644    |
  |   700-799      | 6017    |
  |   800-899      | 5288    |
  |   900-999      | 5109    |
  | 1000-1099      | 5187    |
  | 1100-1199      | 4905    |
  | 1200-1299      | 4369    |
  | 1300-1399      | 8164    |
  | 1400-1499      | 8969    |
  | 1500-1599      | 8430    |
  | 1600-1699      | 8007    |
  | 1700-1799      | 7795    |
  | 1800-1899      | 8143    |
  | 1900-1999      | 9116    |
  | 2000-2099      | 10551   |
  | 2100-2199      | 10261   |
  | 2200-2299      | 7057    |
  | 2300-2399      | 2446    |
  | 2400-2499      | 1225    |
  | 2500-2599      | 1067    |
  | 2600-2699      | 1055    |
  | 2700-2799      | 964     |
  | 2800-2899      | 888     |
  | 2900-2999      | 881     |
  | 3000-3099      | 917     |
  | 3100-3199      | 434     |
  | 3200-3299      | 282     |
  | 3300-3399      | 144     |
  | 3400-3499      | 206     |
  | 3500-3599      | 131     |
  | 3600-3699      | 104     |
  | 3700-3799      | 86      |
  | 3800-3899      | 114     |
  | 3900-3999      | 42      |
  | 4000+          | 5       |
  +----------------+---------+

[noxvilleza]

Really love your chess puzzles, I play Puzzle Rush most days! If you can share, how many puzzles do you add each month? Are they mostly hand-generated or do you have some nifty way of deriving puzzles from real games automagically? Would also be really cool if a puzzle rush spread out puzzle types more it feels like there is heavy clustering of a specific problem type at key rating ranges: early puzzles (1-6) are very commonly back-rank checkmates; 9-12 are discovered check, etc - it would be nicer if the problem meta-tags were not re-used a lot within a single run.

[drw]

Last month we added 2800, the vast majority automatically derived, though we also prize expert review.

We are thinking of doubling that rate next month.

I'm not sure much can be done about the predominance of certain puzzle types in the very early Rush puzzles. Would you prefer more hung pieces?

But in the later levels, mixing it up by themes/tags is an interesting idea, about which I will make some measurements.

[noxvilleza]

Yeah overall some theme diversity. Would also be interested to know the median Elo rating by tag - since that'd serve a good sort of table-of-contents of order for people to learn chess tactics.

What is also pretty interesting would be a (fuzzed) per-tag Elo rating on the user - so they can see which themes you are good/bad at relatively (for example, if you constantly get problems on Trapped Pieces wrong your rating on those problems would be lower than your average Puzzle Rating).

[drw]

Yes, the rating differs by theme quite a bit.

  +-------------------+------------+
  | theme/tag         | mdn_rating |
  |-------------------+------------|
  | Opposition        | 1725       |
  | Passed Pawns      | 1494       |
  | Rooks on Seventh  | 1408       |
  | Pawn Endgame      | 1353       |
  | ...               | ...        |
  | Smothered Mate    | 549        |
  | Stalemate         | 533        |
  | Mate in 2         | 436        |
  | Mate in 1         | 399        |
  +-------------------+------------+

I also like the idea of showing the user's +/- for each theme. Though the data for some themes is better than data for others.

[SamBam]

Why such a large number? Do players really go through thousands of puzzles a month, and remember them all?

If you produced fewer, would that increase the proportion with expert review?

[Nemerie]

Casper Schoppen broke all the previous records in puzzle rush (a mode where you get to solve as much puzzles as possible in 5 minutes) by memorizing most of high rated puzzles. After that chess.com decided to add a lot of new ones.

https://www.chess.com/article/view/puzzle-rush-confirmed-rec... https://www.chess.com/news/view/new-puzzle-rush-chess

Although I don't think there were lots of players who tried to use this tactic among all the ones who occasionally train on chess.com.

[sudhirj]

Ah, thanks. That explains how you manage to have so many available. I like the progressive / adaptive mechanism as well, it keeps the challenge at the right level. Thanks for a great product.

There were some issues (crashes) with the videos on iPad, but I assume you’ve seen that in your logs already.

[drw]

The iOS team doesn't see any recent crashes that associate to viewing videos.

If you want to follow up, feel free to email me your chess.com username and the approximate date when you had trouble; the team could look up crashes which happened to that username.

Both my personal and chess.com email addresses are in my profile.

[bradly]

This is really interesting! Thanks for sharing. What are the average number of moves per puzzle in the top groups? Do they all end up being 5+ moves or can there be 2800+ puzzles with only 2 or three moves?

[drw]

There certainly can be short difficult puzzles, but as I mentioned elsewhere, I think the current 4000-rated puzzles look like outliers and may be removed.

  +----------------+---------+-------------------+
  | ratings_band   | count   | avg_move_count    |
  |----------------+---------+-------------------|
  | ...            |         |                   |
  | 3000-3099      | 917     | 3.9400            |
  | 3100-3199      | 434     | 4.1244            |
  | 3200-3299      | 282     | 4.2766            |
  | 3300-3399      | 144     | 4.7500            |
  | 3400-3499      | 206     | 4.9515            |
  | 3500-3599      | 131     | 4.9237            |
  | 3600-3699      | 104     | 5.0481            |
  | 3700-3799      | 86      | 5.3256            |
  | 3800-3899      | 114     | 6.4035            |
  | 3900-3999      | 43      | 5.9302            |
  | 4000+          | 5       | 1.6000            |
  +----------------+---------+-------------------+

[bradly]

Thank you for sharing the information.

By the way... I really enjoy chess.com. The recent addition of getting to retry your game mistakes is really cool. I'd love to see those kept overtime and be able to retry them all in a random order later.

[drw]

Definitely on the roadmap.

[mactrey]

This is interesting and it makes me really want to see the 5 4000+ puzzles...

[drw]

I am not sure that the 5 specific puzzles maxed out at the 4000 also represent the very best, most fun, and most insightful. Sometimes the reason a puzzle comes up as an extreme outlier is somewhat obtuse, and we are more inclined to remove the puzzle after we see how users interact with it.

For example https://www.chess.com/puzzles/problem/568140 is mostly rated 4000 because ...Qe1+ is so compelling that it pulls in even excellent players, while there's an easy material pickup on the board.

But there are exceptional puzzles throughout the 3000s, like 3500-rated https://www.chess.com/puzzles/problem/769056 , which absolutely anyone can follow, but few can solve on the first try.

[ouid]

why is Qe1+ compelling? It pushes the king to greater safety and activates the rook. If the correct move is e3, then I think that the problem with a 4000 rated puzzle is that the prior is so strong that players feel like they can ignore the obvious moves. You should try just resetting the rating of that puzzle and see it ends up in a different spot.

[drw]

We sometimes do that. I'm resetting that puzzle down to 1480.