[vamin]

You’re setting up a bit of a false dichotomy, in my opinion. Treating something like a black box doesn’t mean you have to “throw up our hands and claim to know nothing whatsoever.” Rather, it means you have to change your approach to understanding the system.

People who are concerned about “black boxes” seem to think that we need a first principles or causal-mechanistic explanation for what’s going on in a machine learning system to have any confidence in it. That couldn’t be further from the case. By interrogating the inputs and outputs of a “black box” you can learn all you need about how it works. Much (if not most) of our understanding of the physical world comes from carefully probing black box systems—systems for which we have no a priori knowledge of mechanism. So, the alternative is not to “throw up your hands,” it is to take a considered, scientific approach to understanding the relationship between inputs and outputs in your model: in what situations it succeeds, in what situations it fails, how changing a single variable affects the output, etc. Yes, that can be difficult for a complex model, but why should anyone expect it to be simple?