[PaulHoule]

I've worked in IR and this has been true about TREC data sets from the beginning and it has also been true about visual data sets. The first step to build a world beating commercial system has been to clean up the garbage in open evals to raise the possible accuracy ceiling.

[dial481]

That's encouraging to hear from someone with IR experience, thanks. Agree completely.