[argonaut]

Humans appear to do quite well on ImageNet (anecdotally, one person got 5.1% error: http://karpathy.github.io/2014/09/02/what-i-learned-from-com...). Of course there are recent deep models that do better than that, but the author opines (and I agree) that an ensemble of trained human annotators would do better than the best deep models.

MNIST is the true toy dataset (doesn't really tell you much about your algorithm's performance) - while there aren't any reported human evaluations of MNIST, LeCun estimates the human error rate is 0.2% - better than any deep models (admittedly without justification: http://yann.lecun.com/exdb/publis/pdf/lecun-95a.pdf).