The algorithms to train, initialize the networks, new architectures are far more important than the hardware advances. If people knew how to train NNs 50 years ago we would live in a different world.
I find it really interesting that when Richard Feynman did a sabbatical at Thinking Machines when they were developing the early parallel execution hardware that's really not worlds away from modern GPUs he got them in touch with one of the leading neural network theorists as an obvious use for the tech. When he wasn't fixing their hardware designs using systems of differential equations.
It would be an interesting thing to know more about.