[yorwba]

This leaderboard tries to prevent that kind of problem:

> The dataset is partitioned into a Challenge Set and an Easy Set, where the former contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. This leaderboard is for the Challenge Set.

Additionally, I don't think they let you try often enough to get a meaningful chance at significantly beating the baseline with just pure randomness.