[zEVE16Ug50tV]

That's a good and deep point. Let's consider the case of writing an algorithm to drive a car. There are some ideas people have had:

1) What really matters is what happens in the worst case, so we need explanations in the worst case, but not necessarily the rest of the time: when the car is choosing a slightly more efficient trajectory, we don't need an explanation of why it 'chose' to do that. In the case of a crash, though, we'd like to know precisely what went wrong. This suggests, perhaps, that we could have a simpler and more transparent fallback system that usually is uninvolved in driving, but that takes 'responsibility' and has the ability to control the vehicle at a minimum level of safety (rather than efficiency).

2) Humans seem to produce make decisions first and explanations later http://www.skepticink.com/tippling/2013/11/14/post-hoc-ratio...: 'To the question many people ask about politics — Why doesn’t the other side listen to reason? — Haidt replies: We were never designed to listen to reason. When you ask people moral questions, time their responses and scan their brains, their answers and brain activation patterns indicate that they reach conclusions quickly and produce reasons later only to justify what they’ve decided.' so perhaps we could also train an ML agent to produce explanations that we find persuasive?

[stareatgoats]

Agree, the need for transparency occurs mainly when something goes wrong. But it is not also a question of control? Without human understanding there is no human control, with no human control we have HAL ... which sort of answers your second point: no, we should not let AI come up with it's own explanations that we just find persuasive.