[nl]

It's easy to say that, in the same way that people now wouldn't be surprised if we could factor large numbers in linear time if we had a functional quantum computer!!

10 years ago no one believed it was possible to train deep nets[1].

It wasn't until the current "revolution" that people learned how important parameter initialization was. Sure, it's not a new algorithm, but it made the problem tractable.

So far as algorithmic innovations go, there's always ReLU (2011) and leaky ReLU (2014). The one-weird-trick paper was pretty important too.

[1] Training deep multi-layered neural networks is known to be hard. The standard learning strategy— consisting of randomly initializing the weights of the network and applying gradient descent using backpropagation—is known empirically to find poor solutions for networks with 3 or more hidden layers. As this is a negative result, it has not been much reported in the machine learning literature. For that reason, artificial neural networks have been limited to one or two hidden layers

http://deeplearning.cs.cmu.edu/pdfs/1111/jmlr10_larochelle.p...

[eru]

Dropout (and maxout) might also count.

[fma]

It's only difficult because no one threw money at it. It's like saying going to Mars is difficult. It is - but most of the technology is there already, just need money to improve what was used to go to the moon.

If you asked people 10 years ago before the moon landing if it was possible, I too would agree it's impossible. But after that breakthrough it opened up the realm is possibilities.

I see AlphaGo more of an incremental improvement than a breakthrough.

[oldmanjay]

It's a basic human bias to believe that anything that you don't have to do (or know how to do) "just needs money" to get done.

[nl]

So are you arguing that superhuman-level performance in just a matter of engineering effort? Or am I missing something?

I'm generally considered to be way over optimistic in my assessment of AI progress. But wow.. that's pretty optimistic!

[argonaut]

I interpreted him as saying superhuman-performance at Go was just a matter of engineering effort, which I wholeheartedly agree with.