https://supercomputersfordl2017.github.io/Presentations/Imag... http://learningsys.org/nips17/assets/slides/dean-nips17.pdf
For the last version of the TPU, Google provided more detail, e.g., in this paper:
https://arxiv.org/pdf/1704.04760.pdf
Hopefully, Google will publish something similar for TPUv2, but I have no knowledge whether or when that might happen.