[david-gpu]

Pytorch relies heavily on the extensive libraries of high-performance kernels provided by NVidia, such as cuDNN.

In other words, it goes something like this:

    Application
    Pytorch (and similar)
    cuDNN (and similar)
    CUDA (and similar)
    NVidia GPU

My opinion, based on what I saw those wizards do, is that reproducing the feature set and efficiency of cuDNN/cuBLAS is deeply nontrivial.