[kqr]

Tell me about it! I work on a search engine powered by relatively basic machine learning of user behaviour. We probably achieve the most relevant results in the world for our customers. That's not the hard part.

The hard part is tightening our development feedback cycle. Since we outperform all competitors, we don't have an oracle to test against. We can automate testing with a small sample of input-output pairs, but the brunt of the work is still done by humans trained and paid to judge the quality of the results. It's an awful position to be in.

I have started looking for better ways of doing it, and the most promising I've found so far is metamorphic testing, mentioned in another comment.

Property testing only takes you a short bit here, as far as I've been able to figure out.

(I have also glanced at the techniques used in bioinformatics, since those guys are good at comparing sequences, but that's more specific to our case and not a general solution.)