We can't explain the implementation details, but a human system can literally explain the logic she used to reach a decision. For example, for applications in the justice system that AI has been recommended for, this is a highly important quality.
Eeeeeeh... what we do is more like parallel construction. We can give a series of plausible steps to explain where we ended up, but sometimes we can't really explain why we did some of the steps.