[markhahn]

GPUs have a large advantage in a very narrow niche: regular and very intensive ops on regular and compact data. ML is not completely ideal (because it's not that compute-intensive), but MC simulations often are. Most things are not ideal for GPUs, especially since it's often not obvious how to nicely scale across GPUs. MPI's strongest point is that it lets you take very good advantage of any topology of general-purpose computers: multicore, NUMA, distributed-memory. Models that emphasize data-parallel programming (co-array fortran, etc) suit GPUs much better. But nothing is going to change the fact that GPU registers are much faster than GPU (card) memory, which is faster than any possible interconnect.