[kiririn]

I think these kind of blind listening tests are fundamentally flawed. For example in the graphics realm (games, video encoding, colour science, etc) all it takes is a momentary black screen between two comparison images to make it vastly more difficult to detect differences. Likewise side by side is also more difficult than swapping between two images instantly. Audio makes it impossible to do an instant swap, at best you’re getting the equivalent of a side-by-side comparison

[maep]

If anything those tests make it easier to find subtle differences, which is good if transparency is the goal. I don't think that makes them fundamentally flawed. They are used throughout the industry, making results comparable.

Of course there are other ITU tests that work without hidden references, looping or even A/B comparison. They require a much bigger listener pool, are more expensive and take longer, thus used less often during development.

[kiririn]

Maybe not fundamentally flawed but audio ABX testing is focused towards short term memory and opinion (especially in unskilled subjects) than I would like. I don't think there is any right answer to audio blind tests.

I'll trust actual validated limits of human perception such as 16/48 audio, 1~3dE colour, etc. And techniques used in video encoding like psnr, ssim, etc are also pretty well grounded in science. Also SINAD

But anything involving a human blindly comparing audio is into audiophile pseudoscience territory, no matter how large a cohort of people or how it is executed

[maep]

I can assure you that audio codec testing is a through science. Tools such as PSNR, PEAQ or POLQA all have limitations and cannot fully replace a human listener. Those familiar with the topic are often vocal critics of audiophile bullshit.

No, this is nowhere near pseudoscience, psychoacoustics is an established field of science.

[mrob]

Audio does not make it impossible to do an instant swap. Any good ABX tool lets you switch between test/reference samples with zero delay. Hear for yourself:

https://abx.digitalfeed.net/list.html

(you can press A, B, or X on the keyboard for instant switching)

[kiririn]

Ye that's what I meant when I said side-by-side. You can't use the same pattern matching that your eyes/brain do when an image instantly swaps, as audio is always temporally moving to be heard at all. Instantly swapping between two audio streams is no better than looking at two images side-by-side