[lars]

Ok. I can give you a 100 million floats, and tell you that if you multiply, add and threshold them in a certain order against a set of pixels, that tells you if the picture is of a cat or a dog.

That is an exact reproducable procedure that tells you how the neural network works. But you are a human being, and have a short term memory capacity of about 7 items. And 100 million parameters is too much for a human to really understand.

The point is that there are strong reasons to believe that no procedure for classifying cat vs dog is small enough that humans can wrap their heads around it. And why is this a problem? The human vision system is exactly the same, complete black box, yet we rely on it every day.

[opless]

No. It isn't exactly the same.

It shares some features, I grant you, but decoding cats/dogs by welding a classifier to the equivalent of V3/V4 isn't what a mammal does.

Furthermore; A "conscious" short term memory of 7-10 sequences is correct. So we break issues down into manageable chunks and it's turtles all the way down.

Comparing the product of >200m years of evolution Vs a decade or so of human endeavour is a strawman.

[naresh_xai]

Think more in terms of function decomposition rather than having to look at each individual parameter and you will find papers and techniques which lead to deep neural networks being quite explainable.

Contrastive LRP would be a good starting point for you for generating high fidelity explanations at any point in the network.

[lars]

I am very aware of these, and I like methods like CLRP. It's great to have these tools to debug neural networks.

However, for many that argue for explainable AI, CLRP falls way short of what they want. In particular, the symbolic AI crowd would scoff at it. This is the crux of the issue in my eyes, that the symbolic AI crowd has taken "explainability" as a way to justify methods that don't work.

I have no issue with methods that allow greater understanding of neural net internals, that's essentially what all neural net researchers spend all their time on (and it's the path towards better performing methods).

[naresh_xai]

Interested in seeing where symbolic AI crowd has disagreed with that. Only group of people who disagree that I know of are a small set of people who think you should build inherently explainable models as opposed to explaining the decisions of deep neural networks.

[salawat]

>Ok. I can give you a 100 million floats, and tell you that if you multiply, add and threshold them in a certain order against a set of pixels, that tells you if the picture is of a cat or a dog.

My friend, you have not even scratched the surface. First off, an elucidation of the inputs and the procedure by which an output is generated does not an explanation of the system make.

When I look at an image classifier, I want to know what features it's using to make a determination of being cat-like. That way, I can compare that with my own experience to make sure if I cut someone loose with a cat/dog detector, someone doesn't get given the idea that a young bear is a dog. Your trivial AI cat/dog detector may identify cat/dog like features in a still, but that's not equivalent with being able to distill the essence of cat/dog from the reality and common experience of the world around us. If you're going to try to sell me on a system that purportedly knows what something is, I expect it to actually represent the level of intelligence you make it out to possess.The neural networks we manufacture are of a level of magnitude so much narrower than what ML people seem to want the lay person to give them credit for.

Think of it this way:

As a programmer, I am expected to be able to create an accurate enough representation of what is going on in a complex system that a non-programmer can connect what the system is doing to whether or not it is doing what it should be. Given enough time and patience on the non-programmer's part, I should be able to transfer and walk through enough information where the non-programmer suddenly becomes a novice programmer because they have had the same foundational skill and knowledge structures communicated to them.

No one will be satisfied with "I chucked this data in, therefore it's a cat/dog detector now. No more questions." Especially when you start applying that to decisions of life-altering importance. You must be 10% smarter than the piece of equipment for it to be lynchpin in a life-critical application. That means being able to explain what your system does, how changes to inputs will effect it, what it's error margins are, what the safe operating conditions are, when it's plain flat out wrong, and as much as possible, why.

Until such knowledge can be sufficiently communicated, I see no reason to take even the most well-known luminary trying to handwave explicability as anything but trying to avoid having to uncover enough of the mystery of what they are working on in order to meet what has been accepted as sufficient due diligence.

To do so is patently unwise, and implicitly accepts far more egg breaking to make an omelette than we (those whose lives will be in the system's hands) should be willing to entertain.

[lars]

> an elucidation of the inputs and the procedure by which an output is generated does not an explanation of the system make.

That was exactly my point. Are we talking past each other?

My point here, if you wish to engage with it, is that when we evaluate trust in an AI system, we care about how good it is. And it is the case that quality is very often anti-correlated with explainability.

Suppose your life depends on winning a game of Go. Would you want AlphaZero on your side, or a Go engine that would present you with a list of the options it evaluated, so you can verify its decision? Of course, AlphaZero would beat the latter program every time.

If this desire for explainability is taken seriously, the result is that we'll end up picking methods that perform worse, and this will cause real world harm as AI becomes a larger part of life critical systems.