[bumby]

Ok, that's a better way to frame it than I was originally thinking. In that context, I'd say 'explainability' is too blunt of an instrument to be used to push back on a model than 'uncertainty'.

IMO, if explainability is the new way to push back on models we're uncomfortable with, it shouldn't be. Uncertainty arguments can be mathematically quantified and defended. Can the same be said for explainability? (Genuinely asking). If not, it's really just a less rigorous way of saying "I'm not comfortable with this model but I can't explain why."

[taeric]

My gut is that it is too easy to make these conversations basically people yelling past each other.

As an example, you are treating uncertainty as a form of tolerance. But you have to explain that, as well. Why is one model 10% uncertain, but another is 30%?

You could just say it is over the data that was trained, but if you can pull it back to used parameters of a model, they may make something more obvious. And it is hard to take uncertainty based on trained data something that transfers to unseen data.

[bumby]

Certainly not yelling, but I’m looking for clarity. Right now it feels like the “explainability” concept is a bit nebulous, even within the NIST document.

Yes, any model is limited by the data used to build it. Relying on that isn’t particularly helpful, just like saying there are unknown unknowns, while true, isn’t helpful. What helps regarding uncertainty, however, is that it can be explicitly defined. Defining uncertainty in parameters is part of the effort; the parameters can be defined within uncertainty as well rather than assuming a point estimated parameter is gospel. That’s one way that helps explain why one model has different uncertainty than another. Some statistical methods, like Bayesian inference, require you to define these assumptions mathematically. All models require assumptions but there’s a world of difference between a black box and one that requires explicitly and mathematically defining them.