[ajuc]

You want repeatable conditions for tests so that you can compare results between models, and against the norms.

When you have repeatable conditions - software in the tested product can detec that and act differently in these conditions.

That's exactly what happened in VW case.

It's nontrivial to fix the test so that it is still repeatable and hard to fool by company determined to fool it.

[madez]

Randomized test with a suitable number of runs will cost-efficiently give useable results.

I agree, it's not trivial. But, it's not hard either.