[sorenbouma]

I'm a TPU user and I'd be interested to see a specific example of something that can be done on TPU but not GPU.

Perhaps I'm just not experienced enough with the programming model, but I've found them to be strictly less flexible/more tricky than GPUs, especially for things like conditional execution, multiple graphs, variable size inputs and custom ops.

[sillysaurusx]

Sure! I'd love to chat TPUs. There's a #tpu discord channel on the MLPerf discord: https://github.com/shawwn/tpunicorn#ml-community

The central reason that TPUs feel less flexible is Google's awful mistake in encouraging everyone to use TPUEstimator as the One True API For Doing TPU Programming. Getting off that API was the single biggest boost to my TPU skills.

You can see an example of how to do that here: https://github.com/shawwn/ml-notes/blob/master/train_runner.... This is a repo that can train GPT-2 1.5B at 10 examples/sec on a TPUv3-8 (aka around 10k tokens/sec).

Happy to answer any specific questions or peek at codebases you're hoping to run on TPUs.

[slaymaker1907]

That doesn't answer the question of what a TPU can do that a GPU can't. I think the OP means impossible for the GPU, not just slower.