> hardware progress is slowing down quickly)
I would be interested to know more about this. I haven't heard yet that the progress of GPUs cores for example is declining quickly ...
The problem is Amdahl's law. You can only parallelize so much. While the brain is certainly extremely parallelized, neural nets do not employ the same algorithms as the brain, and so, unless we find algorithms that are more amenable to parallelization, Amdahl's law is going to get us.
Most modern neural networks implementations are parallelized. And that is why we can run them extremely well on the GPU. For example Volta GPUs delivers 5X increase in deep learning training compared to prior generation NVIDIA Pascal architecture.
This is why I was asking for clarification about the hardware claims.