[axelroze]

The Artificial Neural Network is essentially just a huge set of formulas involving millions and sometimes even billions of values. Debugging in conventional software engineering sense of having a breakpoint and looking at values does not work.

The current state of the art for analysis is ShAP: - https://github.com/slundberg/shap

ShAP is primarily an instance based explainer (one image = one explanation) but if you run it over multiple instances it is possible to gather global model insights on the data. The internals of the model are still quite unexplainable compared to decision trees or anything a human can code (horrible code aside).

There is a group at ETH doing work on adversarial attacks: - https://www.sri.inf.ethz.ch/publications/

While not directly related to explainability their work is on providing bounds on how much can corruption of the input still provide valid output. Very interesting and practically relevant as well.

Finally there is also common sense. If race was a large factor in prediction then the model will implicitly learn to predict races. I am not in medicine and do not know how much it is but if it is then the only way to not learn race prediction is to make race not correlated to the targets.