[huyegn]

I agree with Yann here ... I think the demand for explainability is like a person asking for a "faster horse" when what they really need is a car.

When people ask for explainable models, what they really want (in my opinion) is calibrated and robust uncertainty estimates .

Good uncertainty estimates would let them know when to trust a model's prediction and when to ignore it.

For example, a model trained to predict dog breeds should know nothing about cat breeds, and there should be some way to quantify when it doesn't know!

I've been doing a review of techniques that are becoming more popular in this area:

https://blog.everyhue.me/posts/why-uncertainty-matters/

[ssivark]

While uncertainties are useful, they are certainly not enough. Eg: I can construct adversarial examples which are predicted incorrectly with high confidence.

The underlying reason why high confidence is not enough is that even strong/confident correlations could be misleading when seen in causal light — a black box model trained to predict credit performance might be very confident in rejecting loans for applicants from “poorer” zip codes and approving those from “richer” zip codes — even though those are not actual causes... therefore somebody could exploit the system by renting an address in a rich neighborhood for a couple of months when taking out a big loan (the analogue of adversarial examples).

[huyegn]

I stated that rather than explainability, for most use cases, people just want calibrated and robust uncertainty estimates NOT low quality and uncalibrated uncertainties estimates.

Your example points to models that provide low quality uncertainty estimates, but that's not true for all deep learning models.

I believe it's these low quality uncertainty estimates that lead people to look toward "explainability" as a solution, but for the majority of use cases, I think people just want better uncertainty estimates so that they can "know when they're model doesn't know".

There are techniques now to get higher quality, calibrated, uncertainty estimates that don't suffer from the problems you mentioned and I've outlined these solutions in my posted link above.

Additionally if you're interested, there is some nice recent research from google on the subject:

https://ai.googleblog.com/2020/01/can-you-trust-your-models-...

and from oxford:

https://arxiv.org/abs/1912.10481v1

[ssivark]

Thanks for the links here and above; will take a look.

> calibrated and robust uncertainty estimates NOT low quality and uncalibrated uncertainties estimates.

Could you explain what you mean by “calibrated” and briefly summarize the essential idea behind what allows the learning of robust uncertainty estimates, if not a causal understanding?

If you haven’t already, look up work by Scholkopf, Janzing , Peters and co (over the last decade) for a justification of why causal reasoning is exactly what you want if you want to generalize across covariant earth/dataset shift (which is basically what the Google blog post is about).

[esyir]

Sure, but at the same time, there's the concept of adversarial examples not being the model's fit to noise, but simply being a different, human-imperceivable feature instead.

Considering that ML has access to orders-of-magnitude greater quantities and resolution of information that we do, why do we expect their decision structure to be intensely linked to our own decision process.

[wnoise]

I think that unlike Yann, you're making a good point.

[opless]

I disagree with LeCun but you have a good point. Telling you how much uncertainty there is in a result is a useful metric. But it is only part of what is required (by my definition of explainability).

My counterpoint is when a model trained on dog breeds misidentifies a breed with a high certainty factor, the model should be able to justify (explain) why it made that judgement.

[huyegn]

That's an interesting idea and I get the motivation. I often wonder if we're already able explain predictions. There are things like gradcam to determine which inputs caused which activations. It's just that the explanations aren't very understandable or satisfying from a human perspective.

[naresh_xai]

You should definitely look at advances in the field then. There a lot more promising work beyond grad cam. And there are a few techniques for human oriented explanations - namely TCAV (human oriented explanations) and PatternNet/PatternAttribution as well as contrastive LRP (in terms of much more informative heatmapping techniques which are dependent on intermediate layers as well.)