[TallGuyShort]

In some cases, no. But let's say there was a drug for 99% of people, but totally messed up 1% of people. Well then it's probably really worth understanding how to predict or avoid that. Because the same thing happens with image classifiers. Google had an embarrassing incident where their classifier was very impressive, until it got it wrong and it was really embarrassing. Do you just accept that sometimes image classifiers act racist? Or would it be nice to have a tool that can highlight what parts of the picture contributed most to the classification, so you can identify pictures that would have prevented that error in the training set?