[Melatonic]

I agree with what you are saying but from a philosphical point of view are humans (and intelligence as we define it) also a sort of black box?

Perhaps what would be even better is to document better the process, work, and data that go into making each individual"AI" model. Regardless of whether that AI model is a "black box" or can self explain its behavior we would then have absolute metrics and comparable information to retroactively explain its "decisions". This would not be entirely dissimilar to how we explain individual humans behavior with psychology (although obviously also very different).