[dimatura]

I believe that when the number of parameters isn't in the millions, as in deep networks, there's less of an advantage to using GPUs. But there is definitely research on using GPUs to solve nonlinear least squares optimization problems. Here's one I saw a few years ago: https://dl.acm.org/doi/10.1145/3132188