[jlebar]

V100s are often memory bound when using tensor cores, yes. But I guess my point is broader than that. There is a "right shape" for hardware that wants to excel at a particular workload, depending on the arithmetic intensity, degree of temporal locality, and so on. The point is that you usually can't just turn up one dimension to eleven, it's not usually that simple.

For example, massively increasing the GPU last level cache size would not have the effect of increasing memory bw much on most workloads, because cache only helps when you have temporal locality and gpus like to stream through many GB of data.

This is covered in Hennessy and Patterson if you're curious to learn more. I also talk about it some in the video I linked above.

(Also I doubt that getting rid of f64 support would be a significant die size win. I notice that v100 has, in their marketing speak, twice the fp32 cores as fp64 cores. What do you think are the chances that Nvidia decided a priori this is the optimal ratio? What if instead they are sharing resources between these functional units, at a ratio of two to one?)

To the question of, do you really need fp32 cores, I am not aware of any "widely deployed" GPU model today that does not do significant fp32 work. Perhaps there is research which suggests this isn't necessary! But that is a different thing than we were talking about here, that Nvidia could somehow make a much better chip for the things people are doing today.

I don't want to speak to the question of whether TPUs have f32 hardware, because I'm afraid of saying something that might not be public. But I think the answer to your question can easily be found by some searching and is probably even in the public docs.