[drewda]

Machine learning folks don't know the history of CS or AI, so they've reinvented neural networks as "deep learning"?

Or, industry types are looking for the next big thing, after "big data," and have rebranded neural networks as "deep learning"?

I don't mean to be too cynical, but I still don't understand if "deep learning" represents any meaningful advance besides the ML and EE communities finding the benefits of a certain amount of structure, which is already well established in other lines of research.

[sinwave]

While this complaint generally has validity, their paper [1] does IMO present an advance; it's not just handing a bunch of labeled data off to a large neural network.

IIRC (forgive me, I read the paper a few weeks ago) the solution is at its core a reinforcement learning system, with the deep net only making up the component that predicts reward from a (state, action) pair. With that in hand, there remains the non-trivial RL problem of balancing "exploration vs exploitation" in learning good strategies to play the game(s). While NN's have been used in this capacity before, I believe that, as other comments have mentioned, using a deep net to learn to map a high-dimension state-action space (e.g,the state of the game represented as pixels of the screen at a particular time) to expected reward in real time was indeed an advance, both theoretical and technical.

And, oh yeah, I just remembered that a University of Texas research group is doing work in this area too (there was a recent paper [2] from Peter Stone and others).

(Edited for clarity)

(Edited again to suggest another paper).

[1] - http://arxiv.org/pdf/1312.5602.pdf

[2] - http://www.cs.utexas.edu/~pstone/Papers/bib2html-links/TCIAI...

[eivarv]

Deep learning is not just neural networks, but rather the application of these in deep (i.e. many-layered) architectures, broadly speaking.

This enables hierarchical learning of increasingly complex concepts – building new concepts upon less complex concepts from previous layers. Deep architectures are thus able to learn high abstractions, as in [1], for instance.

If you have not yet done so, I would strongly urge you to read some papers on the subject from the last decade (e.g. Hinton, Bengio or LeCun), or even just skim through the Wikipedia entry [2].

[1] http://www.technologyreview.com/view/532886/how-google-trans...

[2] http://en.wikipedia.org/wiki/Deep_learning

[return0]

Deep learning is a large scale application of Restricted Boltzmann Machines, of which Hinton (among others) was a pioneer. But that was in the 80s, not in the 2000s.

http://en.wikipedia.org/wiki/Restricted_Boltzmann_machine

[eivarv]

I don't believe the term "deep learning" is restricted to RBMs only – at least that's not the way I've seen the term used in literature (e.g. Deep Convolutional Neural Networks, various deep Autoencoders, etc.).

[return0]

Convolutional networks were also developed in the 80s as well as backpropagating algorithms (autoencoders). The way i see it used, "deep" usually means many layers, indicating a difference in quantity, not in quality.

Point is, the science was there since the 80s, and not much has changed.

[eivarv]

Sure, but these types of deep architectures haven't really been practical until relatively recently.

Well, then we're in agreement about the meaning of the term. Deep Learning, then, would be Machine Learning using any of these deep architectures – be they Restricted Boltzmann Machines, or otherwise.

[Houshalter]

I sometimes wonder if in the 2030s, people will be complaining about how all the interesting stuff was really invented back in the 2010s.

But yes, the available computing power has been a huge limitation for much AI research.

[tormeh]

It's just that previously neural networks with more than 3 layers have been prohibitively expensive to train. Now that we've discovered some shortcut tricks and got the training on to GPUs we can finally have those neural networks with lots of layers. Only the implementation details have changed.