[ebalit]

They weren't any recent breakthroughts. But LSTM and ConvNet are breakthroughts. It just took a lot of times to prove it.

[bonoboTP]

Convnets follow pretty naturally from multilayer perceptrons. Perhaps backpropagation was a breakthrough, enabling the training of ANNs on data, instead of hand-tuning.

But the idea of neural nets is very old, going back to Rosenblatt and connectionism.