[kxyvr]

Many already do, but not necessarily explicitly. If an optimizer accepts user defined functions for its evaluation and derivatives, these computations can be done using a GPU even though the optimizer itself knows nothing about the GPU. For example, GPU are extensively used in parameter estimation problems associated with PDE contrained optimization. Essentially, the PDE solves use GPUs to solve the differential equation and then then results are fed back into the optimizer. Many of these packages use common open source optimizers.

More generally, there's a question of where the algorithms themselves benefit from GPUs or parallelism in general. For large scale nonlinear, continuous optimization problems using second-order, Newton like methods, the big costs are in the function evaluations, their derivatives, and the linear system preconditioners/solves. Generally speaking, how the function evaluations and derivatives are computed are on the user. For the preconditioning/linear system solves, there's value in parallelism. However, here, the GPUs have traditionally lagged. Basically, we need a factorization, be it sparse or dense, and it's only been recently where good library support has been extended for GPUs. For the longest time, the entire matrix factorization needed to fit onto the GPU and many of these matrices were large. That said, for optimizers that accept a user-defined preconditioner, the use of GPUs is already possible.