[cleverpebble]

I definitely enjoyed reading your article!

Did you play around with any AI-specific accelerators (eg TPUs?)

Looking at some basic cost analysis from a stranger on the Internet - https://medium.com/bigdatarepublic/cost-comparison-of-deep-l... - you can probably get a decent price reduction in training, especially using preemptive instances (and perhaps a better pricing contract with Google/AWS)

It's kind of crazy how the shortage of GPUs is affecting pricing on physical devices. My RTX Titan I bought in 2019 for $2,499 runs almost $5k on Amazon and is in short supply. The Titan V options you linked (although I think theres a typo because you referred it it as a Titan X) is an option - but it is still super overpriced for it's performance. Of course, this will probably settle down in the next year or two, and by then there will be new GPUs that are ~2-4x flop/$ compared to the V100/A100.

[kettleballroll]

At these sizes, tpu would definitely be the way to go, and would likely be a lot cheaper (and potentially faster) than GPUs.

[Learnedvector]

Last I've checked (a year or two ago) PyTorch support for TPU's were atrocious. Have they gotten any better?

[tubby12345]

PyTorch XLA is mature backend. In fact several other accelerators support PyTorch by lowering from XLA.

[ypcx]

https://github.com/pytorch/xla/