[51109]

As we start to use AI/ML for more tasks, the need for model interpretability rises. We expect doctors to explain their gut feelings, much like we expect computer vision models that detect disease to explain their findings and have a (theoretically sound) estimate of confidence.

SVM's were so popular, pretty much because they had a firm theoretical basis on which they were designed (or "cute math" as deep learners may call it). As Patrick Winston would ask his students (paraphrasing): "Did God really meant it this way, or did humans create it, because it was useful to them?". Except maybe for the LSTM, deep learning models are not God-given. We use them because, in practice, they beat other modeling techniques. Now we need to find the theoretical grounding to explain why they work so well, and allow for better model interpretability, so these models can more readily be deployed in health care and under regulation.

[_0w8t]

The article calls for an explanation not why some ML method works, but for an explanation of a particular ML result, like why a car drives this way or why a patient got a cancer. While I hopeful for the former, I just do not see the basis for the latter.

If some regulations shall require such explanation, the end result will be fake stories like parents tell to the children that Moon do not fall because it is nailed to the sky.

[51109]

I don't think the paper asked for that. Relevant quote:

> machine learning is more concerned with making predictions, even if the prediction can not be explained very well (a.k.a. “a black­box prediction”)

So in your example: an algo may explain that a car slows down, before taking a turn, because else it would likely crash. It may even get to a threshold ("under these weather conditions, anything over 55Mph is unsafe when taking a turn of such and such degree"). Statistics can help with that.

Welling is not asking for deep learning models to explain why a person got a cancer, but to explain its reasoning when it diagnoses a person with cancer ("I am confident, because in a random population of a 1000 other patients, these variables are within ..."). Statistics can help with that. It aligns with their mind set and tool set.

Regulations are cheated even with these kind of explanations, but that is for another story (black box models may provide some plausible deniability).

[_0w8t]

I am referring to this fragment:

> Thus, for many applications, in order to successfully interact with humans, machines will need to explain their reasoning, including some quantification of confidence, to humans.

No doubtful there are cases when an explanation is easy. Often this is because we have a very solid model like physics of a car. In fact since we know the model, we do not need an explanation, we must demand that the algorithm follows the model or declare it unfit.

But how can we expect an explanation for a behavior in a critical situation on a road that was not explicitly programmed and when the algorithm decided to turn to a particular degree bases on a non-trivial inference? Similarly, when an algorithm decides if a patient needs an emergency operation or if they can wait, why can we expect an simple explanation especially for the patient with rare conditions when algorithm again must perform an inference, not a deduction from 1000 very similar cases?

[jupiter90000]

Maybe you couldn't always expect an explanation, but having one would certainly be useful. Methods that could get the job done while also providing enough information to explain or at least shed some kind of light on why the solution was chosen would likely be preferable.