[tansey]

The contrastive divergence paper that Hinton published in 2006 definitely set the field off again. I remember entering grad school in 2010 and everyone was still really excited about using unsupervised pretraining. However, nowadays no one uses it.

It just turns out that with GPUs and stochastic gradient descent, no one needs any of that stuff. There are some tricks out there to making it really work, though. In that sense, Hinton's dropout paper has probably had a longer lasting effect on the field.

But either way, I doubt what OP is saying will be true. None of the real advances in deep learning are coming from self-taught coders in the middle of nowhere. They're coming from big labs with lots of resources, both physically and intellectually. This stuff takes a lot of hard thinking by a lot of people who understand optimization and probability. It also takes a ton of compute power and massive datasets, which won't be available to a hobbyist.

[soared]

While a hobbyist doesn't have a team of experts (though forums could replace that), right now they do have access to massive data sets and cheap computing resources. There are tons of huge free data sets and cloud and hardware are cheaper than ever.

[anonyfox]

I honestly do not think that DL is the answer. It's just a special use case of NN with multiple layers, and NNs itself are just one school of machine learning, IMO not even the most promising one.

[_mhr_]

What is the most promising one?

[argonaut]

The irony is that dropout is increasingly and often viewed as unnecessary in recent deep learning work.