[moseandre]

>> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it.

I have to be able to debug anything I work with; debugging is necessary.

>> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop"

It's important to look into why the stop sign was mis-classified, even if the classifier is a subcomponent.

[gmueckl]

I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).

[darepublic]

It seems to me that understanding these deep learning models will be very tough since the reason why ml was necessary in the first place was the sheer complexity of the solution being too much for generations of programmers to solve

[gmueckl]

Understanding an existing solution is very different from trying to find it. We get taught things at school that seem simple and obvious to us now, but were extremely hard to discover or invent.

I see these training approaches as a tool that makes machines search the space of potential solutions for certain types of problems more efficient than humans. The next logical step is to try and understand these solutions and improve upon them.

Consider adversarial input. Right now we cannot tell which classes of adverserial inputs can exist for a given NN. We can only try to find representative examples. If you had a good enough understanding of how the NN works internally, you have a chance to derive the full set of adversarial inputs or - maybe -prove their absence.