[JyrkiAlakuijala]

Psychovisual image quality tests are difficult and complicated. Selection of the population, methods, guiding text, motivation, etc. can lead to strange things happening.

As an example of the difficulties an academy leading corpora TID2013 has a reversal in quality between categories 4 and 5 -- highest quality images seem to get slightly worse quality ratings than the next highest quality images.

I observed a leading image quality laboratory to produce substantially different results for the same codec when the person conducting the experiments changed.

There are hard opinions if images should be reviewed one image pixel to one monitor pixel, or if images should be zoomed like they are in practical use, particularly on mobile.

Should zooming be allowed? Should people be allowed to move closer to the monitor if they want as part of the viewing? etc. etc.

[vanderZwan]

Yes, I agree it is incredibly hard. On the other hand, if anyone has put in the work to give the "least bad answer so far" to this question it's Jon Sneyers.

I don't have time to dig through Twitter just now, but instead of just picking one image metric they've used all of them to also compare the testing methods themselves.

[JyrkiAlakuijala]

All my own experience and observations also suggest that Jon did an excellent job on this and is likely the first person to ever do it well with crowd-sourcing. In the past I have seen similar quality of results with hand-picked experts (image quality people, photographers etc.), but not with volunteers or crowd-sourcing.