[Homunculiheaded]

Feature extraction can be aided by unsupervised data but will certainly work with labeled data. One of the advancements bundled under 'deep learning' is how we can leverage unlabeled data (which is much easier to come by) to improve performance. And of course you can always do unsupervised learning with labeled data, just toss out the labels ;)

It's actually the multiple layers hidden units that perform non-linear feature extraction and the unsupervised pre-training is simply a means to do this better (theoretically, although we don't really know what's happening as much as it would seem).

Most of the current research shows deep neural nets to be state of the art in image classification and nlp. I don't know that it is the case that deep learning techniques do not work out side this area, it's just there hasn't been much published on it either way. Although I do believe the Kaggle Merck contest was neither of these, and deep learning out performed all other techniques http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it...