[fauigerzigerk]

I think explanation is itself a rather complex concept. At what point do we consider something as explained? Usually it has to do with identifying some causal factors and their relationships so that we can intervene and explore counterfactuals. But in many cases we are forced to act on the basis of incomplete explanations (e.g. in medicine).

I think there will be regulation that requires some users of AI to provide an explanation upon request. For instance, banks could be required to "explain" why you didn't get that loan. What if the decision is based on a credit score that includes some AI prediction that ultimately relies on the entire training corpus?

The bank can give you a list of factors that play into the decision but they may not be able to explain deterministically why a very similar customer did get that loan. At that point I think we're going to resort to statistics that prove a lack of bias against certain protected characteristics, but that's not really an explanation, is it?

I think we will never get useful and complete explanations for everything that AI does. Society will just accept some explanation-like thing or proxy and move on.