[visarga]

You can't make a full on model audit on a neural net, but there are architectures such as the Transformer (an attention scheme) that can give a lot of insight into what the neural net thinks. We can also visualise what inputs maximally activate a deep neuron in a CNN. Not all DL models are truly black boxes.

[AstralStorm]

The problem is the models have no reflection capabilities unlike people. The explanation is always done by a really different external system and sometimes by an actual intelligence.

State models (Markovian) are sometimes able to explain things but not always really, especially in complex cases.

[visarga]

On the other hand, humans tend first to take a decision and later retrofit an explanation to it, even if it is completely wrong, and we assume the explanation is the reason and not the effect.