[modeless]

Man, why are all the silicon people so fixated on spiking nets? Maybe in 20 years once we figure out how the brain works they'll be great but if you built a convnet chip instead then it could be smashing records in real problems like speech recognition, translation, image identification, etc, today. Where are the convnet chip startups?

[p1esk]

First of all, what makes you think a hardware convnet will perform better than a software convnet?

Second, you can implement a convnet with a spiking circuit: http://link.springer.com/article/10.1007%2Fs11263-014-0788-3

[modeless]

That paper does not implement training, only testing. Low-power testing is good to have e.g. for mobile applications, but it doesn't increase the capabilities of our algorithms. What we need to advance the field is faster training of larger nets. That's where the really interesting applications will be discovered.

A convnet-optimized chip could clearly be much faster and more power efficient than a convnet running on a CPU or GPU. The move from CPU to GPU brought a 10x speedup already, but GPUs are hardly ideal for running convnets. For one thing, they have tons of graphics-specific hardware that's useless for convnets and could just be deleted in a convnet chip. For another, GPUs are much more flexible than necessary for convnets. The main operation you need to perform is convolution and you could make fixed-function convolution units that would be much more power and area efficient than generalized GPU shader cores. For yet another thing, there's no reason to believe that 32-bit IEEE 754 floating point is the best power/precision tradeoff for convnets. I'm willing to bet that you could go much lower. You could even experiment with approximate arithmetic; 0.5 ULP precision is probably not necessary.

[p1esk]

Building a hardware convnet is only beneficial when you figured out the exact parameters of the network. Everything is hardwired. Therefore, it's useless if you want to experiment with lots of different parameters, tricks, or architectures to "advance the field".

Moreover, building such a chip is an expensive and long process, and given how fast GPUs are improving, it's not clear that by the time you build it, it will still be competitive.

Finally, if you want to speed up training, try to figure out a better algorithm. For example, humans can learn from very few training examples. Current neural networks need many thousands, or even millions. There's a potential million-fold speed up in training time right here - and to find it, you need the flexibility of the software.

[sushirain]

Maybe the brain can do one shot learning only after it has trained for years?

[p1esk]

Maybe. We want to reduce the amount of supervised training. I think that's what Hinton is trying to do with his capsules.

[sz4kerto]

Partly because they want to do fundamental research, not R&D. This means they don't necessarily have a well specified set of problems where they'd need neural networks, but they want to understand biological neural nets. One approach in doing so is trying to stay as close to real neural features as possible.

[p1esk]

Actually the reason (at least in this paper) is that spiking nets are much more power efficient.