[maep]

If anything those tests make it easier to find subtle differences, which is good if transparency is the goal. I don't think that makes them fundamentally flawed. They are used throughout the industry, making results comparable.

Of course there are other ITU tests that work without hidden references, looping or even A/B comparison. They require a much bigger listener pool, are more expensive and take longer, thus used less often during development.

[kiririn]

Maybe not fundamentally flawed but audio ABX testing is focused towards short term memory and opinion (especially in unskilled subjects) than I would like. I don't think there is any right answer to audio blind tests.

I'll trust actual validated limits of human perception such as 16/48 audio, 1~3dE colour, etc. And techniques used in video encoding like psnr, ssim, etc are also pretty well grounded in science. Also SINAD

But anything involving a human blindly comparing audio is into audiophile pseudoscience territory, no matter how large a cohort of people or how it is executed

[maep]

I can assure you that audio codec testing is a through science. Tools such as PSNR, PEAQ or POLQA all have limitations and cannot fully replace a human listener. Those familiar with the topic are often vocal critics of audiophile bullshit.

No, this is nowhere near pseudoscience, psychoacoustics is an established field of science.