[lars]

> an elucidation of the inputs and the procedure by which an output is generated does not an explanation of the system make.

That was exactly my point. Are we talking past each other?

My point here, if you wish to engage with it, is that when we evaluate trust in an AI system, we care about how good it is. And it is the case that quality is very often anti-correlated with explainability.

Suppose your life depends on winning a game of Go. Would you want AlphaZero on your side, or a Go engine that would present you with a list of the options it evaluated, so you can verify its decision? Of course, AlphaZero would beat the latter program every time.

If this desire for explainability is taken seriously, the result is that we'll end up picking methods that perform worse, and this will cause real world harm as AI becomes a larger part of life critical systems.