| I have to disagree with some of your points. > The main difficulty is high dimensionality. We simply do not have mathematical frameworks to analyze structures in the 100s of dimensions. The main difficulty in deep learning is the non-convexity of the optimization problem. We can handle simpler problems in high dimensions just fine. The oracle complexity bounds for projected gradient descent in convex optimization even hold for infinite-dimensional problems - see work of Nesterov. Most of the hard questions about deep learning remain hard even for neural networks with low-dimensional inputs, outputs, and hidden layers. Also, some of the more fruitful approaches in deep learning theory involve taking the limit as the width of one network layer goes to infinity. > For some reason, theoretical computer science seem to contribute quite little to the practical deep learning world, while primarily concerning itself with the complexity theory and computability questions. Fun stuff, but... Somebody needs to do this. Lots of theoretical researchers are trying to figure out why deep learning works. Check out the work of Jason Lee, Simon Du, Sebastien Bubeck, etc. Most of these researchers have a CS background. |