[bumby]

Certainly not yelling, but I’m looking for clarity. Right now it feels like the “explainability” concept is a bit nebulous, even within the NIST document.

Yes, any model is limited by the data used to build it. Relying on that isn’t particularly helpful, just like saying there are unknown unknowns, while true, isn’t helpful. What helps regarding uncertainty, however, is that it can be explicitly defined. Defining uncertainty in parameters is part of the effort; the parameters can be defined within uncertainty as well rather than assuming a point estimated parameter is gospel. That’s one way that helps explain why one model has different uncertainty than another. Some statistical methods, like Bayesian inference, require you to define these assumptions mathematically. All models require assumptions but there’s a world of difference between a black box and one that requires explicitly and mathematically defining them.