[bonoboTP]

Machine learning people basically agree that there weren't any big breakthroughs in deep learning. The success and the hype is mostly a combination of more computing power and more data. The algorithms (convolutional neural network etc.) were invented back in the 1980s and even earlier.

There have been some improvements but they are incremental indeed. More use of ReLU, dropout etc. But it's not a new paradigm at all.

[ebalit]

They weren't any recent breakthroughts. But LSTM and ConvNet are breakthroughts. It just took a lot of times to prove it.

[bonoboTP]

Convnets follow pretty naturally from multilayer perceptrons. Perhaps backpropagation was a breakthrough, enabling the training of ANNs on data, instead of hand-tuning.

But the idea of neural nets is very old, going back to Rosenblatt and connectionism.