Old time neural networks not even came close to current models in terms of performance.
I clearly remember in my machine learning class, one professor mentions neural networks and says it is slow and impossible to tame when the layers goes up thus loses its popularity. That is just 3-4 years ago.
I took some machine learning classes a while 1-2 years ago at the TU Berlin, and while there was some emphasis on neural nets, it was just as much emphasis on SVMs, with the remainder of the time filled with other traditional models. The lecturer had done some research into SVMs, though, so it was probably just bias rather than old-fashionedness. He was also of the mathematical sort, and I suspect the unanalyzable nature of NNs rubbed him the wrong way. I'm not saying that all other AI/ML methods should be abandoned, but NN definitely didn't get the attention it deserved. Deep learning techniques weren't covered at all, for example.
Deep learning has really only recently become successful with new learning algorithms such as constrastive divergence and convolutional neural networks. Previous efforts were focused around backpropagation, but due to the signal loss across many layers there was never enough information in the output layer to successfully train the network.
Deep learning has really only recently become successful
hinton coined the term "deep learning" around 2006/2007 (more around deep belief nets/RBMs, but still, same thing), if that's considered "recently."
constrastive divergence and convolutional neural networks.
CD was also ~10 years ago. CNNs were reading your checks and postal zipcodes in the mid 90s.
successfully train the network.
In the early 90s, RNNs were driving cars on highways using only webcams under basically VFR. No giant sensors, no LIDAR, no GPS, no mesh networks, just camera input.
The 2012 ImageNet results were what really launched the current interest.
Before that there was little evidence that any form of neural network was massively better than other forms of machine learning. Now it has become clear that isn't the case.
Regular neural networks were popular until they stopped delivering best-in-class results for a lot of problems.
Recent hardware made it possible to train more layers and they're now getting cutting-edge results in many areas again so they're now getting more attention again.
Yes and No. Layer to layer training of deep belief nets were doing very well before alex.net came along, although alex.net was seminal for images. People often miss the big 'advances' in RNN's, although those go back to the 90's and earlier. A lot of this (but not all) is also due to more data, better and faster hardware (including GPU's). Although there have been some important algorithmic enhancements too.
I clearly remember in my machine learning class, one professor mentions neural networks and says it is slow and impossible to tame when the layers goes up thus loses its popularity. That is just 3-4 years ago.