[mwytock]

Many methods in machine learning and statistics benefit from two types of theory:

1) Optimization theory - which says, if I repeat this iterative method N times I will find a (nearly) globally optimal solution

2) Statistical theory - which says, if I observe this process N times I can accurately estimate a population quantity with high probability

Deep learning does not benefit from the same theoretical guarantees.

For the most part the response from the community is "but it works really well!" which is a fair and valid response especially since what most practitioners care about is predictive accuracy.

Personally, I find applying neural networks extremely annoying at times due to the amount of twiddling of hyperparameters, slow convergence, etc.