[jruohonen]

Fair enough, and that's why I suppose for instance CATPCHAs and such make us do silly free labor.

But generally, hilarious labeling errors are widespread already in benchmarks:

https://news.ycombinator.com/item?id=26628778

Companies also seem rather carefree with their labeling, including even in contexts in which accuracy is paramount:

https://news.ycombinator.com/item?id=38455338

And things get interesting when you start labeling alleged political bias, for instance:

https://news.ycombinator.com/item?id=35982799

Thus, in general, I'd appreciate research on the "accuracy" (i.e., labelers' inter-group agreement, etc.) of labels used in the wild.