[pero]

you can buy at least 2 nanopi neo2's for every rpi3b - they have gbe and a non-gimped soc, and you can fit in at least 2 for every rpi, maybe even 4

there are also more powerful alternatives of the same footprint and at same or slightly cheaper price point (rock64)

[jandrese]

Ugh, it has one of those godforsaken Allwinner chips on it. good luck getting 750 of those working at once. The RPis are more expensive up front, but they don't require the researchers to learn Mandarin to get tech support or read the docs, on the outside chance such things are even available.

[kortex]

Engineering teams rarely trifle over price of parts when reliability and confidence are greater concern. Sure, the neo2 is cheaper, and the rock64 is more powerful, but I haven't heard of either. What is the probability of encountering a weird not-before-seen bug on these platforms, vs Pi? Now what's the cost of dealing with that bug, including the risk-cost of having to scrap the entire platform because it's unscalable? It's probably at least $20x750 = $15k.

Pi has at least 10x more community eyes crawling over the whole system than the next three combined. You can't put a price tag on that.

[ballen]

Also consider the ecosystem for these boards. Do you want your researchers dealing with the non-upstreamed support around Allwinner chips? Additionally consider time to solution, this machine is based on BitScope's Blade: http://bitscope.com/product/blade/.

[ballen]

32-bit vs 64-bit is largely inconsequential when the nodes only have 1-2GB of RAM. The ecosystem I'm referring to means you can run a vanilla Linux on a RPi with no extra work. You can google a problem and have a reasonable chance of finding a solution around the RPi and so on.

[jandrese]

From what I understand you can see a decent improvement in speed by going 64bit on ARM because they used the crossover to dump a fair bit of legacy braindamage in the instruction set that was preventing them from making certain optimizations.

On the other hand, I look at the 3000 core figure and think that it's roughly on par with high end GPUs. The clock rates aren't terribly different either. The range of applications where this beats out GPU solutions is probably fairly narrow, especially given the terrible IO bottleneck on the RPis.

For comparison, a $7,500 TITAN X has 3072 CUDA cores clocked at 1Ghz. This cluster has 3,000 CPU cores clocked at 1.2Ghz. On the TITAN card all of those cores share the same 12GB of memory with 336.5GB/s of memory bandwidth. On the cluster every 4 cores shares 1GB of memory with (I think) 3.6GB/s bandwidth. Of course communication outside of those 4 cores is restricted to 0.0125GB/s at best.

[pero]

no it's not inconsequential

for one, why are _researchers_ using largely obsolete technology; for another, many high performance computing tasks perform significantly faster on 64bit (e.g., lmdb)

[pjc50]

Performance isn't actually the objective of the Pi cluster; the people using it have a real supercomputer next door. It's a testbed so they can validate programs before transferring them to the expensive supercomputer.

[pero]

is their real supercomputer 32bit? because if it's not then i'm not sure how they are validating anything

[kortex]

I would imagine going from a 10-node to 100-node system is more overall complicated than going from 32 to 64. Sure the instructions change, but that should basically be all abstracted away by the toolchain. However job management, allocation, data logistics, queues, cache invalidation, bottlenecks, etc, are all key issues that compound non-linearly with scale.

[kortex]

Chemists still use thin-layer chromatography, a technique 100 years old, day-in-and-out, in every lab in the world, even when HPLC and NMR exist. Why? It's cheap, fast, and works well enough.

[pero]

if we are 'considering the ecosystem' then that means these are running a 32bit os, as mentioned in my original comment