[PeterisP]

Can you elaborate why you think that Moore's law is considered dead? For me it seems that the general progress for the computing hardware in question (GPUs and specialized ASICs, not consumer CPUs) we're still seeing steady improvements in transistors/$ and flops/$ and expect it to still continue for some time at least.

[aszen]

Yes specialized hardware for AI are seeing steady improvements, I'm curious if these improvements rely on the particulars of the algorithms running on these machines. As an example several of the AI chips use lower precision floating point numbers than general CPUs since the algorithms in use for training nns don't need the higher precision.

I actually wonder if having specialized AI hardware isn't the same problem as having specialized AI models, that is in the short term it will improve efficiency but in the long run prevent discovery of newer general learning strategies because they won't run faster in existing specialized hardware.