[sillysaurusx]

Re: PyTorch TPU support, has anyone checked it out beyond "it works"?

There are many aspects of TPUs that I'm not convinced are easy to port: Colocating gradient ops, scoping operation to specific TPU cores, choosing to run operations in a mode that can use all available TPU memory (which is up to 300GB in some cases), and so on.

These aren't small features. If you don't have them, you don't get TPU speed. The reason TPUs are fast are because of those features.

I only glanced at PyTorch TPU support, but it seemed like there wasn't a straightforward way to do most of these. If you happen to know how, it would be immensely helpful!

As far as predictions go, AI will probably take the form of "infinite remixing." AI voice will become very important, and will begin proliferating through several facets of daily life. One obvious application is to apply the "abridged" formula to old sitcoms. (An "abridged" show is when you rewrite it using editing and new dialog, e.g. https://www.youtube.com/watch?v=2nYozPLpJRE. Someone should do Abridged Seinfeld.) AI audio as already made inroads on Twitch, where streamers like Forsen allow donation messages to be read off in the voice of various political figures (and even his own voice). The Pony Preservation Project was recently solved with AI voice (https://twitter.com/gwern/status/1203876674531667969) meaning it's possible to do realistic voice simulations of all the MLP characters with precise control over intonation and aesthetics.

Natural language AI will continue to ramp up, and people will learn how to apply it to increasingly complex situations. For example, AI dungeon is probably just the beginning. I recently tried to do GPT-2 chess (https://twitter.com/theshawwn/status/1212272510470959105) and found that it can in fact play a decent game up to move 12 or so. AI dungeon multiplayer is coming soon, and it seems like applying natural language AI to videogames in general is going to be rather big.

Customer support will also take the form of AI, moreso than it is already. It turns out that GPT-2 1.5B was pretty knowledgable about NordVPN. (Warning: NSFW ending, illustrating some of the problems we still need to iron out before we can deploy this at scale.) https://gist.github.com/shawwn/8a3a088c7546c7a2948e369aee876...

AI will infiltrate the gamedev industry slowly but surely. Facial animation will become increasingly GAN-based, because the results are so clearly superior that there's almost no way traditional toolsets will be able to compete. You'll probably be able to create your own persona in videogames sooner than later. With a snippet of your voice and a few selfies, you'll be able to create a fairly realistic representation of yourself as the main hero of e.g. a Final Fantasy 7 type game.

[nl]

I did try training a Pytorch BERT-derived model on TPUs (on Colab) and it didn't work out-of-the box. (Where didn't work=was using the CPU)

I didn't dig into it to find out why.

[zak]

(I'm one of the Cloud TPU product leads)

We've seen multiple BERT-related PyTorch models training successfully on Cloud TPUs, including training at scale on large, distributed Cloud TPU Pod slices.

Would you consider filing a GitHub issue at https://github.com/pytorch/xla or emailing [e-mail] to provide a bit more context about the specific issue you encountered?

Here's the current PyTorch/TPU troubleshooting guide, which provides information on how to collect and interpret metrics that are very helpful for debugging: https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.m...

Thanks!

[riku_iki]

> BERT-related PyTorch models training successfully on Cloud TPUs

How do you see it? Do you look at your client's code?

[nl]

Google wrote BERT and they provide technical support to the FB Pytorch TPU port so it's not entirely surprising. RoBERTa, (Fb's variant) would be a good candidate to test it with.

[zak]

We only see code when customers open-source it or otherwise explicitly share it with us. We are directly in touch with several customers who are using the PyTorch / TPU integration, so we hear feedback from them, and we also run a variety of open-source PyTorch models on Cloud TPUs ourselves as we continue to improve the integration.

[PeterStuer]

Minor remark:

"With a snippet of your voice and a few selfies, you'll be able to create a fairly realistic representation of yourself as the main hero of e.g. a Final Fantasy 7 type game. "

Extrapolating from FB/Instagram I would wager a 'fairly realistic' representation while technically an option will be shunned by most in favor of a 'fairly idealized' representation or alternatively some sort of myopically polarized extreme trait representation.

[jeffshek]

I agree with so many points here. Generative AI is one of those things that are immediately applicable. Instead of problems that require 100% accuracy, generative models don’t require that. As long as the avatar looks close enough, it works. And no one really needs explainability there.