[throw_away_777]

You can do the same with deep learning, your model will fail when the train data and test data are systematically different. This isn't about understanding the model, it is about understanding the data. Understanding the data is much more important than interpreting the model. Systematic uncertainty is a problem for all models.

[hatmatrix]

Yes - but how do you know when your test data and training data are different in the way that the model cares? With conventional methods, you can assess similarity by distances in the feature space of the model, or you have a physical understanding of why the model works than you have a better intuition about the differences in data that are important.