[taeric]

The article makes a decent discussion piece. Such that it does seem that both are pitched as panacea cures for why the models sometimes don't work.

Combined with the idea that folks think models would be better used if they presented their uncertainty, I can see the direct line to models needing explainability before we deploy them.

To that end, why do you think "no?"

[bumby]

Not the OP, but maybe you can help me understand the relationship.

As I understand it, uncertainty is a statement of risk. Explainability is statement of understanding how a system works to produce an outcome. None of the four NIST principles seem to conflate the two.

I can say I understand how my brakes on my car may fail to work because it's an explainable mechanical system with known failure modes. However, that's different than the statement about the uncertainty that the brakes will work as intended. In the latter, there is a statistical probability that gets translated to a risk statement. I think one needs to have an explainable system in order to arrive at an uncertainty risk statement. They are both related to quality, but speak to different aspects of the problem.

[taeric]

You are just highlighting that they are different things. The article seems to be pointing out that they are now getting used for the same reasons/aims.

That is, yes, they do ultimately tell you different things. But, per the article, both can be used to push back on using a model.

That is to say, in prior years, folks pushed back on models for them not presenting their uncertainty. Seems there is a growing push to push back if they do not present explainable reasons.

[bumby]

Ok, that's a better way to frame it than I was originally thinking. In that context, I'd say 'explainability' is too blunt of an instrument to be used to push back on a model than 'uncertainty'.

IMO, if explainability is the new way to push back on models we're uncomfortable with, it shouldn't be. Uncertainty arguments can be mathematically quantified and defended. Can the same be said for explainability? (Genuinely asking). If not, it's really just a less rigorous way of saying "I'm not comfortable with this model but I can't explain why."

[taeric]

My gut is that it is too easy to make these conversations basically people yelling past each other.

As an example, you are treating uncertainty as a form of tolerance. But you have to explain that, as well. Why is one model 10% uncertain, but another is 30%?

You could just say it is over the data that was trained, but if you can pull it back to used parameters of a model, they may make something more obvious. And it is hard to take uncertainty based on trained data something that transfers to unseen data.

[bumby]

Certainly not yelling, but I’m looking for clarity. Right now it feels like the “explainability” concept is a bit nebulous, even within the NIST document.

Yes, any model is limited by the data used to build it. Relying on that isn’t particularly helpful, just like saying there are unknown unknowns, while true, isn’t helpful. What helps regarding uncertainty, however, is that it can be explicitly defined. Defining uncertainty in parameters is part of the effort; the parameters can be defined within uncertainty as well rather than assuming a point estimated parameter is gospel. That’s one way that helps explain why one model has different uncertainty than another. Some statistical methods, like Bayesian inference, require you to define these assumptions mathematically. All models require assumptions but there’s a world of difference between a black box and one that requires explicitly and mathematically defining them.