[nerdponx]

"it is not very good at explaining the process it has gone through to reach such a conclusion".

To be fair, generating human-understandable explanations of predictions in a complicated nonlinear model is difficult in general. You typically need to come up with some kind of simplification of the model (like LIME [0]), and it's far from perfect or generally-applicable.

[0]: https://homes.cs.washington.edu/~marcotcr/blog/lime/

[soraki_soladead]

SHAP is more recent and does work with (just about) any ML model: https://github.com/slundberg/shap

There's still more work to do in interpretability but models are rarely opaque black boxes with no way to see inside.