[dwohnitmok]

An amusing point from the comments

> It’s not even trying to be competitive, it’s just guessing how the game will continue. If you blunder, it might guess that this must be a game between two blundering fools, and play accordingly.

In a certain sense, GPT-2 is optimized to "look good to people interested in AI." Above all else it tries to generate plausibly-human-looking things, while completely oblivious of any other goal. This makes it an interesting fit for scenarios with objective scoring criteria. It may never be "good" at the scenario, only entertaining to human observers.

[maxander]

It’s a pity that the common annotation for a “surprising” move is to follow the move code with an examination mark (or two) rather than preceding it; otherwise we would have a simple way of making the model generate surprising moves on command. :)

[drusepth]

You could always train another model off reversed move notation and wrap it with something that reverses the output again. ;)

[lowdose]

Then you probably make a rookie mistake and assume every opponent counters your actions with a fully functioning system 2 in Kahneman speak.

[awb]

> otherwise we would have a simple way of making the model generate surprising moves on command

One method would be to just take the 3rd or 4th best move option. It wouldn't be a winning strategy but it would probably be pretty surprising and still moderately effective.

Also, I'd imagine that the most surprising moves would require the opponent to make a very precise series of moves to avoid a losing position. While the best moves usually slowly improve your position, the most surprising moves might polarize the position by giving the opponent a chance to improve but also making the chance of a costly blunder more likely.

In that sense you could look for moves that give the opponent the fewest number of positive expected value moves rather than finding moves that give your opponent the lowest possible EV response.

[closeparen]

Plenty of human behavior is generating plausibly-human-looking things completely oblivious of any other goal.

[lowdose]

90%.