[bloak]

I don't think you can detect "bias" by running "a bunch of tests". "Bias" is a very slippery concept and is probably essentially subjective. When people say an algorithm is "biased" what they seem to mean is that when the judgements of the algorithm are compared with the judgements of a committee of fair-minded and diligent humans then the number of positive outcomes for members of some fashionable minority that we care about is less that what it was with the human judges. It's hard to automate that. And in any case, if you manipulate the algorithm until it "passes" a test like that then you might not really have improved it: when you turn a measure into a target it ceases to be a good measure.