[kenjackson]

The ironic thing is that in the HPC space, where they've done large scale message passing for decades -- the holy grail has always been large scale shared memory!

Now admittedly the programming models in HPC were ugly (MPI), but nevertheless the lack of shared state and the use of message passing certainly didn't make it easy to write high performance parallel apps.

I think the problem is fundamentally hard. And when a problem is fundamentally hard the solution to it often is "that other thing we haven't really tried yet". Until you've really tried it.

[nostrademons]

A lot could just be that the requirements are very different between HPC and typical message-passing business apps. In HPC, you want to squeeze every ounce of performance out of the cluster, by definition. If you could get rid of the message-passing overhead, that would be a huge speedup.

For most business processes, you don't care about performance all that much, you just want things to be easy to change. Message-passing works fairly well for that: it's easy to understand, composable, and let's you swap out one component for another as long as the interfaces are compatible.

BTW, message-passing isn't exactly untried. It's the basis for the service-oriented architectures that underlie Google, Amazon, FaceBook, and many other large businesses. It works very well for that problem domain.

[kenjackson]

The message passing issue with HPC codes wasn't perf overhead of the messages. Rather it was that developing message passing applications was very complicated. But message passing apps, when written correctly gave very good performance.

With that said, you're right. If you don't care about performance, or if you have very large grains of computation then message passing is relatively easy (although so is just about any model with those requirements). The question is what about when you actually do care about performance and your grains aren't so large that doing communication half-way across the planet isn't acceptable? When I'm trying to get 60FPS in my physics engine, I probably don't want to use a web service interface.

[gruseom]

That's very interesting. Can you elaborate? I assume your point is that the HPC people were forced into message-passing by the lack of large-scale shared memory, but would have preferred the latter because they found it easier to program for? And that this somewhat contradicts our fashionable ideas about how to write concurrent programs?

Also, is their situation analogous to multicore today? i.e. does many GB of RAM shared by many cores count as "large-scale shared memory" (or does it not, e.g. because of cache effects)?

[kenjackson]

Pretty much what you describe is my point. That is why the common paradigm in HPC was that you used OpenMP on a node (shared memory model) and MPI across nodes. It's not difficult to do MPI on a node, but nobody wanted to do that. And there were several attempts to put OpenMP on clusters. The most recent one I know of being Intel (http://cache-www.intel.com/cd/00/00/28/58/285865_285865.pdf).

I think it is still a little analogous. The main takeaway is that message passing doesn't make things easy. I've spent many of days debugging message passing applications. You often trade-in one type of problem for another. If anyone is interested in more detail, I can go into an example or two.

[gruseom]

Yes, please go into an example or two.

[kenjackson]

So here's a typical problem that I'd have in an HPC application. I'd have some space, represented by some 3D structure (maybe an array, or even an object-based particle system). I need to do some computation over this space -- often using some type of stencil, so in order to compute the value at <x,y,z> I need values of coordinates some distance from x,y,z.

The part that often ends up being tricky is the fact that I need to send data from processor A to processor B. And I want to send as little data as possible. So one of the first sources of bugs is that when I do my gather-scatter I make a mistake mapping a value to a coordinate. In shared-memory you never have to do this mapping back and forth, so its not an issue.

Next issue is related to the fact that I don't want to ever block waiting for data. There are a variety of models for handling this. I can do a non-blocking receive, and do some work waiting for the data to arrive. This is often another source bugs as people will often do work that depends on the new data, but they chug along without it. Add the new data when they get it, and alas their computation is already hosed.

And the last common error in this case is handing the data off to the wrong object (or processor) or being confused as to which data you're receiving at any given point in time.

Now all of these can be handled by simply being careful, and using some good programming practices. But they are just simple, if not grossly naive, examples of issues you have with traditional message passing that don't exist in shared memory.

[gruseom]

Interesting; thanks.

What you're describing sounds to me like the complexity of ferrying data around and scheduling computations is being offloaded to the app programmer. Presumably the intent behind things like OpenMP on clusters is to take care of all that behind the scenes and let the user pretend that it's all shared and program accordingly. Is that correct? If so, how far would you say such distributed infrastructure has gotten to date? Is it usable for real work, or do people end up having to learn so many limitations and workarounds that they're no better off than programming against the lower-level model in the first place?

Another question: even when there is shared memory you still have to coordinate the various processes that are operating concurrently on it so they don't clobber each other, and that, as everyone knows, is complicated too. So there is a tradeoff here. It sounds like your point is that given a choice, the HPC community would rather program against shared memory using traditional concurrency mechanisms (threads, locks, etc.) than deal with the complexities of the alternatives. Am I reading you correctly? If so, that's a pretty major point which suggests that the general-purpose programming community may be gearing up for a wild goose chase.