[pie_hacker]

The match between Stockfish and AlphaZero was played with certain unjustified parameters (time control, ponder off, different hardware, no opening book or endgame tablebase for Stockfish etc.). By "unjustified," I mean that the authors of the paper did not justify their choice of parameters in the paper as being designed to implement a fair match.

At a glance, the parameters of the match seem unfair to me -- and tilted heavily towards AlphaZero. If the code, were open source, this would not matter; anyone could run a rematch. As it is, I haven't seen any convincing evidence that AlphaZero is stronger than Stockfish when Stockfish is allowed to use its full breadth of knowledge and run on equal hardware.

[yesenadam]

There has been a rematch recently vs Stockfish, with a couple of hundred games. AlphaZero won 155-6! [0] There are fascinating videos with grandmasters commentating on some of the games. They're played in an exciting, sacrificial, swashbuckling style, nothing like any other top computer engine, and it seems that may affect the play of top (human) players for the better. e.g. see

Matthew Sadler on chess24

https://www.youtube.com/watch?v=JacRX6cKIaY&list=PLAwlxGCJB4...

Daniel King

https://www.youtube.com/watch?v=pFtY7gNRVRI&index=3&list=PLh...

[0] https://en.wikipedia.org/wiki/AlphaZero#Chess_2

[fairplay2]

The Wikipedia link says this was against Stockfish 8. Could people please stop spreading FUD here?

If there's no public tournament, this might as well not have happened. I do not understand why Google is always special. Other engines are open, Google can test against Stockfish but not vice versa.

All these web companies take, take, take from Open Source and rarely give back.

I'll write a paper now that I beat Carlsen, but I'll refuse to do so in public.

[yesenadam]

>Could people please stop spreading FUD here?

Do you mean me? If so, why not say so. "Could people please stop X-ing here?" is very passive-aggressive. I looked up 'FUD' - Fear, uncertainty and doubt. Not sure how what I said counts as any of those. Your comment certainly seems to want to spread FUD, however. (And why is this your account's only comment on HN?)

I don't know the significance of your first sentence either. I know nothing about the versions of Stockfish used for this or anything else. Maybe you're taking too much for granted. i.e. that I know enough of the minutiae to understand your comments. Could you fill in the dots a bit? And who are 'all these web companies'?

I'm not super-interested in AlphaZero (or computer chess generally) - haven't read any of the papers, for example. But there's a lot of talk about the various Alphas in the online chess world, since before it was playing chess, and I found these videos very impressive. And it's ridiculous to say "This might as well not have happened". Maybe true for you, but not for the chess world, at all.

[thom]

So, after the first match in which they played against an obviously crippled version of Stockfish, and published games that involved blundering full queens worth of material, I too was skeptical of the second paper. However the setup was entirely reasonable (they basically copied what had recently been used in a computer chess championship and it was plenty beefy) and no weird time controls were used.

You can still run the PGN of the published games back and find some places where Stockfish 8 will analyse its own moves and find them blunders, so it _would_ be good for all this to happen out in the open, but I don't think there's any large scale deception going on here. I think it's beyond reasonable doubt now that AlphaZero is easily as strong as they are claiming.

[thom]

A lot of the moves most praised by GMs are seen as the only moves in the position by Stockfish 9/10. I think there's a huge amount of cognitive dissonance going on, so that people can label AlphaZero's play more 'human'.

Anyway, I wouldn't be surprised if AlphaZero lines have existed at the top of the game for some time. Would be a no brainer for someone to have made Google an offer after the first paper.

[mda]

So if Alpha zero trashes Stockfish 10 will you be saying, Oh Stockfish 11 sees these as only positions.. Please.

[thom]

No, I will just be questioning the humans that label these moves as unique snowflakes that no other type of engine could conceive of.

[mda]

I watched almost all the games with commentaries, I agree with the commentators, the style is definitely unique and not seen before. If the engine could see these they would not get absolutely destroyed.

[yesenadam]

I don't know what you mean by:

I think there's a huge amount of cognitive dissonance going on, so that people can label AlphaZero's play more 'human'.

or:

if AlphaZero lines have existed at the top of the game for some time.

[thom]

I think people are overly keen to imbue AlphaZero with characteristics they associate with human play. Stockfish plays lots of alien lines, often relying on perfect defense, that no human would probably attempt in a real game. It's painful and jarring to admit that perfect chess might make no sense to humans, even after the fact. So I think people have emotional reasons to want to call AZ's play more human-like and natural, because it redeems human chess intuition. And to an extent it's valid, AZ is strategically much more sound as you'd expect from it's brilliant evaluation of positions, but it is still powerful enough to play completely inscrutable lines and to accept positions that look plain ugly. And besides, Stockfish also plays some beautiful stuff and newer versions play many of the nice moves we're seeing praised in YouTube videos. Just my cod psychology take, not really a deep point, and not something I'd really argue strongly about.

The second point, I'm saying it's likely some teams, possibly Caruana's have had access to AlphaZero already, and if the public had better access to its analysis we might be able to look at more games from 2018 and see its insights pop up. Certainly some of the lines Caruana played at the World Championship match were strongly backed by AlphaZero in the videos released, but weren't Stockfish's choice, for example.

[yesenadam]

Thank you. :-)

[noir_lord]

That was the first paper.

Second paper they played fair even giving stockfish a 10 to 1 advantage on same hardware as tcec uses.

[thom]

You can still run the games past the exact commit of Stockfish they used and it finds blunders in its own play, so it still feels like there's a lack of transparency. But I don't think anyone strongly believes AlphaZero isn't the best at this point.

[mda]

Like which move? I wonder why stockfish developers do not claim any of this?

[thom]

You can checkout the exact commit of Stockfish from the paper and perform the analysis of the published games yourself. I doubt the Stockfish developers have bothered because it's an old version.

Can I just state again, because of your aggressive tone in multiple comments now, that I do believe AlphaZero is stronger, I don't believe there are real shenanigans going on, but it's _still_ sad that we can't reliably, publicly verify this stuff.

[mda]

You are the one making this claim: "You can still run the games past the exact commit of Stockfish they used and it finds blunders in its own play, so it still feels like there's a lack of transparency."

This claim implies foul play, I asked for a shred of evidence. Call me aggresive if you want, I just can't stand this kind of bullshit.

[thom]

It's not bullshit, you can download the version of Stockfish and ask it to analyse the games! I did this with scid, you can too. Around move 27 in game one is one example. I don't intend to repeat the tedious process because my curiosity is satisfied, and you're just being obnoxious.

[thom]

(And I know in the past, before reading the paper, I've assumed they were cheating this time because they made so little effort the first time around, but I'm happy to admit I was wrong).

[ag1]

glinscott said "We need a public exhibition match to settle the score, ideally with some GM commentary." above.

As for the GM side, Nakamura basically said the same.

[mda]

Still, zero proof of foul play. But sure i would very much enjoy watching Alphazero destroy Stockfish in a public match.

[thom]

Yeah, I'm certainly not claiming AlphaZero isn't the stronger engine (probably by quite a substantial margin). The first match _was_ a pretty clear fix, but the second paper is an entirely reasonable setup, so I have no real suspicions about the process on an ongoing basis, just a shame nobody can prove it for themselves.

[vkerner]

Paper is patient. We won't know until AlphaZero participates in a fair tournament, like Leela, which recently came in third after Stockfish.