[a1369209993]

> but rather that people at the time didn't think large models with many parameters would effect meaningful change. This was even true three years ago, albeit on a different scale.

I've also noticed this, and want to ask: who are these people? Do they not have (~80-billion-neuron) brains? (And that's neurons, with by most estimates thousands of synapses each; so you're actually talking on the order of tens to hundreds of trillions of neural network parameters before you reach parity with biological examples.)

[marmakoide]

In the early 2000's, it was believed that the topology of a neuron network was a major factor to get it to work well, and that throwing more neurons and computing power alone would not suffice. In a sense it was not wrong : convolutional nets were an early example of neuron network topology that enforced translation invariance while being parsimonious in tunable parameters.

An other factor was that SVM were all the rage back then, because they had nice math and fitted the computational resources of a contemporary workstation.