[vanderZwan]

Yes, I agree it is incredibly hard. On the other hand, if anyone has put in the work to give the "least bad answer so far" to this question it's Jon Sneyers.

I don't have time to dig through Twitter just now, but instead of just picking one image metric they've used all of them to also compare the testing methods themselves.

[JyrkiAlakuijala]

All my own experience and observations also suggest that Jon did an excellent job on this and is likely the first person to ever do it well with crowd-sourcing. In the past I have seen similar quality of results with hand-picked experts (image quality people, photographers etc.), but not with volunteers or crowd-sourcing.