That paper does not implement training, only testing. Low-power testing is good to have e.g. for mobile applications, but it doesn't increase the capabilities of our algorithms. What we need to advance the field is faster training of larger nets. That's where the really interesting applications will be discovered.
A convnet-optimized chip could clearly be much faster and more power efficient than a convnet running on a CPU or GPU. The move from CPU to GPU brought a 10x speedup already, but GPUs are hardly ideal for running convnets. For one thing, they have tons of graphics-specific hardware that's useless for convnets and could just be deleted in a convnet chip. For another, GPUs are much more flexible than necessary for convnets. The main operation you need to perform is convolution and you could make fixed-function convolution units that would be much more power and area efficient than generalized GPU shader cores. For yet another thing, there's no reason to believe that 32-bit IEEE 754 floating point is the best power/precision tradeoff for convnets. I'm willing to bet that you could go much lower. You could even experiment with approximate arithmetic; 0.5 ULP precision is probably not necessary.
Building a hardware convnet is only beneficial when you figured out the exact parameters of the network. Everything is hardwired. Therefore, it's useless if you want to experiment with lots of different parameters, tricks, or architectures to "advance the field".
Moreover, building such a chip is an expensive and long process, and given how fast GPUs are improving, it's not clear that by the time you build it, it will still be competitive.
Finally, if you want to speed up training, try to figure out a better algorithm. For example, humans can learn from very few training examples. Current neural networks need many thousands, or even millions. There's a potential million-fold speed up in training time right here - and to find it, you need the flexibility of the software.
A convnet-optimized chip could clearly be much faster and more power efficient than a convnet running on a CPU or GPU. The move from CPU to GPU brought a 10x speedup already, but GPUs are hardly ideal for running convnets. For one thing, they have tons of graphics-specific hardware that's useless for convnets and could just be deleted in a convnet chip. For another, GPUs are much more flexible than necessary for convnets. The main operation you need to perform is convolution and you could make fixed-function convolution units that would be much more power and area efficient than generalized GPU shader cores. For yet another thing, there's no reason to believe that 32-bit IEEE 754 floating point is the best power/precision tradeoff for convnets. I'm willing to bet that you could go much lower. You could even experiment with approximate arithmetic; 0.5 ULP precision is probably not necessary.