Did you read the last part? Pytorch uses drivers, and drivers exists for Google's TPU and Apple's M1 GPU as well and both works pretty well. I have tested both and it reaches similar MFU as Nvidia.
Maybe on a particular model/dataset but extremely unlikely in general. Again, like another commenter pointed out: if you truly believe it isn't that hard we would love to hire you at Meta ;)
Yes some operations are not supported in MPS/TPU and falls back to slower CPU. But for common architectures like transformers and convnets, it works very well for all the datasets.
I never claimed it was easy. I meant in my opinion it is in the order of 10s of millions dollars of investment, not a trillion dollar CUDA moat that people comment here.
Are M1 GPUs available for data center deployment at scale? Are Google TPUs available outside of Google? Can Amazon or Microsoft or other third parties deploy them?
Anyone that wants off the shelf parts at scale is going to turn to Nvidia.
And because the world buys all the shelf stuff from Nvidia and uses CUDA, Nvidia get the largest debugging user base there is. This way Nvidia can continue to make CUDA even better and cycle repeats.