I don't understand why A.I. can't be explainable. Can't they just add logging every time it makes a decision and then trace through the trail of decisions to the final result?
The AI that is not explainable is not because it cannot log things, its because the semantic interpretation of what it can log is hard. Starting with the real world input (which we understand) a lot of algorithms progressively apply mathematical transformations till reaching the output. It is the real world "meanings"of these transformations, or what is eventually learned: the stack of these transformations - that is hard to grasp.
If you are willing to accept that as an explanation. Yes, it's easily explainable. A big list of anonymous decisions is easy. Naming those decisions is hard.