[PaulHoule]

Depends on what you are labeling. How accurately could you look at pictures of cars and tell marked police cars from regular cars? When I do them in a hurry though I fat finger maybe 1-5% of them so it takes some cleanup.

[jruohonen]

Fair enough, and that's why I suppose for instance CATPCHAs and such make us do silly free labor.

But generally, hilarious labeling errors are widespread already in benchmarks:

https://news.ycombinator.com/item?id=26628778

Companies also seem rather carefree with their labeling, including even in contexts in which accuracy is paramount:

https://news.ycombinator.com/item?id=38455338

And things get interesting when you start labeling alleged political bias, for instance:

https://news.ycombinator.com/item?id=35982799

Thus, in general, I'd appreciate research on the "accuracy" (i.e., labelers' inter-group agreement, etc.) of labels used in the wild.

[canaus]

How much more useful would it be to get multiple people to label data?