[david-gpu]

> The real moat that nvidia has is their hardware capability, and CUDA is the disguised moat.

There is an inmense amount of work behind the cuDNN libraries that outsiders keep ignoring.

These sort of high performance kernels are co-developed in very close collaboration with the hardware architects designing the chip. Speaking of the hardware in isolation of the high performance libraries reveals a deep misunderstanding of how the system was built. This is true of any mature vendor, not just Nvidia.