Hacker News new | ask | show | jobs
by bjwbell 4088 days ago
Too bad he didn't talk about GPGPU killing MPI too or not. I don't know enough to say.

I'm not familiar with the HPC space but I thought a lot of new work, at least in machine learning, was migrating to GPGPU instead of traditional CPUs. The compute per $ or per watt payoff is too large to ignore.
3 comments
stephen272 4088 days ago
I wouldn't say GPUs are killing off MPI. You still need some way to pass data between nodes/GPUs (most of these datasets can't fit within a single GPU). What you are seeing though, is less and less use of the CPU. If code developers decide to use GPUs, they try and move their data onto the GPU and keep it there for as long as possible (data movement across PCIe is a killer for performance). ORNL's new machine Sumiit [1] will have 1/4 the nodes of their current machine, Titan, but multiple GPUs per node. Thus emphasizing the move away from CPUs and onto GPUs. Again though, there are still 3K nodes and you need some efficient way to pass data between those nodes.

[1] https://www.olcf.ornl.gov/summit/

link
markhahn 4088 days ago
GPUs have a large advantage in a very narrow niche: regular and very intensive ops on regular and compact data. ML is not completely ideal (because it's not that compute-intensive), but MC simulations often are. Most things are not ideal for GPUs, especially since it's often not obvious how to nicely scale across GPUs. MPI's strongest point is that it lets you take very good advantage of any topology of general-purpose computers: multicore, NUMA, distributed-memory. Models that emphasize data-parallel programming (co-array fortran, etc) suit GPUs much better. But nothing is going to change the fact that GPU registers are much faster than GPU (card) memory, which is faster than any possible interconnect.
link
ljdursi 4088 days ago
I think with GPGPU, the issue is a little different; it's local computation, so a bit orthogonal to distributed-memory coordination. But it is interesting to see how many higher-level libraries and other tools (like OpenMP4/OpenACC) are springing up around GPU usage. It's hard not to be a bit jealous...
link