Completely. In fact whenever regulators are involved (for example on anti money laundering), any black box model is a no go, precisely due to that inexplicability.
Maybe a "parallel reconstruction" method would allow to identify the top reasons why a neural net made a decision and repackage those as the explanation.
Or run an ablative test where part of the features are removed and their effect measured, so as to identify which features drive the decision most.
Or run an ablative test where part of the features are removed and their effect measured, so as to identify which features drive the decision most.