[Animats]

Yes, "high performance computing" is dying. There's no commercial market for it. Check the list of the top 500 supercomputers in the world.[1] The top 10 are all Government operations. In the top 25, there are a few oil companies, mostly running big arrays of Intel Xeons.

CPU clock speeds maxed out between 3-4GHz a decade ago. Nobody develops special supercomputing CPUs any more. The market is tiny. Old supercomputer guys reminisce about the glory days when IBM, Cray, Control Data, and UNIVAC devoted their best R&D efforts to supercomputers. That ended 30 years ago.

Supercomputers have poor price-performance. Grosch's Law [2] stopped working a long time ago. Maximum price/performance today is achieved with racks of midrange CPUs, which is why that's what every commercial data center has. Now everybody has to deal with clusters of machines. So cluster interconnection has become mainstream, not the province of supercomputing.

[1] http://www.top500.org/list/2014/11/ [2] http://en.wikipedia.org/wiki/Grosch%27s_law

[andrewla]

The article is somewhat confusingly written (or maybe I was just confused by "killing it" colloquially meaning "doing well"), but even a cursory glance shows that it is not arguing that non-distributed high performance computing is relevant. I'm not sure what your post is responding to.

It is instead arguing that traditional HPC is being made irrelevant because traditional HPC uses MPI (the first successful distributed/parallel computing library), which is increasingly irrelevant in favor of newer libraries for the same task.

[wffurr]

Did you RTFA? It's about MPI:

"MPI is a language-independent communications protocol used to program parallel computers."

Runs fine on commodity clusters.

[spott]

>Runs fine on commodity clusters.

Kind of.... For simple, low communication jobs this is true. But when you start trying to find the eigenvectors of a large sparse matrix, communication becomes your bottleneck, at which point MPI on commodity clusters (those without a really fancy interconnect) "works", but not fast enough to be useful.

[vonmoltke]

I don't think "really fancy interconnects" makes a cluster not commodity, since the original post in this thread is about supercomputer processors. You can put Infiniband in any system with a PCI-e 3.0 bus.

[dhaivatpandya]

Communication would become a bottleneck regardless of whether you're using MPI or something else. The problem you're talking about has nothing to do with MPI; it is intrinsic to distributed computing itself.

[art187]

And what system on commodity clusters is fast enough in this case?

[peterwwillis]

What would you define as a "commodity cluster"? To me it's a 512-core vendor-specific blade server with special interfaces to get more bandwith at lower latency across longer links. But maybe i'm just an old fogey.

[markhahn]

blades were never more than a marketing trick: the offer nothing that can't be achieved in a standard chassis. there were a few multi-chassis SMP/NUMA machines that had cache coherency over external interfaces, but that was neither commodity nor HPC.

[sgt101]

1) Not vendor specific 2) Not blades 3) 10Gbe, not special interfaces

That is a commodity cluster.

[coldtea]

Did you fucking read TFA?

It's title literally is: "HPC is dying, and MPI is killing it".

His comment shows more understanding of the article's main point (about the demise of HPC) that your "it's about MPI"...

[gaius]

CPUs any more

CPUs, err, schmee-PUs. It's all about the interconnect and people can and do make special interconnects.

[acadien]

Yeah the 5 dimensional torus network used at MIRA is just too cool not to bring up here. Modern supercomputers are increasingly become less and less discrete.

https://computing.llnl.gov/tutorials/bgq/

[darkmighty]

What's a 5 dimensional torus? I know a 3D is Circle x Circle, is a 5-d torus a Circle x Circle x Circle x Circle?

If so, that could be interpreted as simply a 4D square grid which wraps around the edges, right? (just as a 3D torus is a 2D grid which wraps)

[acadien]

Think of it as loops in 5 dimensions (x,y,z,a,b). I believe each node connects to 10 different neighboring nodes (2 in each of 5 dimensions), although I could be mistaken on that. You can actually tune which direction you prefer the nodes to communicate over by passing certain flags when you submit a job.

Also here's an image... which I admit is not terribly useful, but its what the national lab people put out. https://computing.llnl.gov/tutorials/bgq/images/5Dtorus.400p...

[darkmighty]

Oh so it's actually a 5D grid that wraps around. Confusingly that would make the standard torus a 2D torus, they're referring to the surface dimension instead of the euclidean space it can be embedded into.

[yellowapple]

> CPU clock speeds maxed out between 3-4GHz a decade ago.

Even for x86 this isn't true (4+GHz is at least possible), let alone platforms like POWER which have already pushed beyond 5GHz. Fancier things like vacuum-channel transistors, graphene transistors, etc. could push that even further once they break into commercial viability.

Not that clock speed alone really matters all that much compared to the other performance benefits of high-performance RISC architectures like POWER and SPARC...

> Nobody develops special supercomputing CPUs any more.

Today I learned that Blue Gene was a figment of my imagination :)

Special supercomputing CPUs are still being developed. The reason why they seem insignificant is because their market size has remained relatively constant, while the markets for general-purpose, non-supercomputing-specific platforms have grown much more rapidly. This doesn't mean supercomputing is dead necessarily, just like how the invention of the microwave oven doesn't mean that ordinary ovens are suddenly dead. Rather, it's just an indicator of different use cases, and the different markets thereof.

> The top 10 are all Government operations.

It's a bit misleading (though I suppose technically accurate) to list academic institutions (like the University of Texas, which holds the #7 spot) as "Government operations"; they're government-funded, yes, but there's a big difference between that and, say, an actual government agency directly managing such an installation. I also fail to see how even a majority of those being government installations has anything to do with anything; governments typically have much greater capital to spend on such things - and greater need for such things - than all but the most massive commercial entities.

HPC was never really the purview of commercial enterprises anyway (unless they had extreme computational requirements). The uptick in the use of COTS products for high-performance computing among enterprises (particularly big Internet-reliant ones like Google) wasn't really at the expense of the HPC crowd losing potential users; it's rather just something that formed very recently alongside HPC already being a niche topic.

Basically, by your arguments, "high-performance computing" has been dying for basically as long as it's existed.

> Grosch's Law [2] stopped working a long time ago.

Only because the world switched to clustering, where Grosch's Law doesn't quite apply, and hasn't addressed the limitations of current transistor technology (like the above-mentioned vacuum-channel and graphene transistor technologies, among many others).

> Maximum price/performance today is achieved with racks of midrange CPUs, which is why that's what every commercial data center has.

That's what "every commercial data center has" (this isn't exactly true, but we'll go with it for now) more because of price alone than because of an actually-calculated price/performance ratio. Businesses tend to think in terms of short-term investments much easier than they tend to think in terms of long-term investments (in contrast with academic and often government institutions, which tend to think in the opposite direction, and therefore have entirely different sets of problems in many cases).

Meanwhile, the big businesses that really do actively calculate an optimal price/performance ratio (like Google) aren't the ones using COTS solutions; they usually have the financial capability to invest in homegrown solutions and cut out any unnecessary expense, and are certainly not just buying a bunch of prebuilt servers from Dell. Google in particular has started to invest heavily in IBM's Open POWER initiative, probably due to a perception that POWER will offer a better price/performance ratio than x86 in their already-very-customized hardware stack.

[dalke]

"Today I learned that Blue Gene was a figment of my imagination" .. and I learned that Anton doesn't exist either.

[WallWextra]

NEC is still making their weird vector computers. I have no idea who buys them, but there's a hotchips about it.

[carlsborg]

These days Crays run intel Xeons.