Hmm, I was trying to do the same with KS test on performance data but it seemed extremely sensitive to outliers even when an eyeball test of the two distributions look near exact. Have you ran into any of those issues?
Not with outliers, no, I think it handles them particularly well (conservatively.) But yes to small consistent differences (e.g. uniform 1%) that affect the relative order of results but not by an very important amount. So have to consider effect size even when the test statistic is strong.