[backpropaganda]

I take issue with precisely the notion that deep learning models are "black-box". They're pretty transparent, and just because people haven't gained adeptness in it yet says more about its cutting-edginess than about its interpretability.

The average programmer at a tech company won't be able to tell us how a particular complex piece of code works, but that doesn't stop us from building complex software.

Deep learning methods are also not off-the-shelf type algorithms. Using them does require knowledge of the domain. This doesn't fit with the "black-box" narrative.

In fact, SVMs and DTs are black-boxes due to their off-the-shelf nature. (jk lol)

[ganfortran]

I think the black-box nature or argument of deep learning lies in the fact of its parameters, not the architecture. Sure, the operations of deep learning, what they do is defined ahead-of-time, but together we still miss the point why it works under some scenarios but fails in another.

Take the recent discussion on reddit/ml for example, people are still debating about whether it should be conv-bn-relu or conv-relu-bn. This is a pretty widely used building block, if not the most widely used one, however, people still don't understand why the latter could work or even outperform the former in a lot applications since it filters out all negative values thus destroying/skewing the underlying distribution for bn. And for BN alone, there is a lot of questions to ask, like the running statistics feels like a hack, however it works very well in reality.

So I take no issue of calling deep learning nowadays a black box. We are far, very far from understanding why this monster does this well in solving so many problems. That is why it is interesting. Some researchers' attitude is confusing to me, because apparently there is a big juicy problem out there, waiting to be cracked, yet, they are distancing themselves away from it.I cannot help thinking it is out of contrarian, that the fear what they have worked for so long may not be useful after all. But true researchers should feel excited for the opportunity to be able to participate when the theory is still vanilla and contribute to it.

[backpropaganda]

Your example about the debate of BN usage demonstrates that it is possible to look inside a deep network and debate. That we don't know the answers doesn't mean the answers don't exist, or are impossible to find, which is what the term "black-box" suggests.

Of course, more research in tools for model interpretation would be awesome, and my own lab has done a lot towards it, and this remains an important topic. More is desired, but what we have right now is pretty good too, and is not at all inferior to old-school methods, esp. considering the performance.

[lightcatcher]

I'd argue that a neural net is "black-box" in the sense that nobody really can give a coherent answer to "what happens if I perturb/double/negate this parameter" where the parameter might be deep in some weight matrix. Maybe this isn't a useful question because of the distributed representations within neural nets, but it is at least an answerable question for other models.

Do you know of any work on interpreting neural nets that are being used for non-image tasks?

[throw_away_777]

The deep learning buzz-word applies to models which learn feature interactions in a fairly complicated manner. While it is possible to explain what a model is doing, the model is a black-box in the sense that without a computer you could not develop the model. The model is making a lot of decisions, and explaining why all those choices are made is not really feasible. In contrast, I could create a linear model by hand easily, it just won't be as good at complicated predictions. However I think the stigma against black-boxes is undeserved. The human brain is a black-box and no one argues against using it.

[ansgri]

The human brain is a black-box and no one argues against using it.

Well, nobody's against using the brain, but the current trend in most subject domains is to avoid overrelying on intuition, and checking and supporting the conclusions using structured thinking methods, i.e. logic.

That is, deep learning is an intuition, and you have to have a high-level explanatory/verification mechanism that would support or reject the answer.