[gorkish]

The AXI bus internal to the Pi's SoC is only capable of about 4gbps, and it carries DMA, so ~2gbps is more or less the hard limit for any kind of combined IO operation like disk<=>network no matter what kind of hardware you use for disk and network.

So yes, each pi can easily saturate its own 1gbps interface, so a system like ceph that parallelizes reads and writes among nodes is severely crippled by the onboard switch choking off bandwidth to external clients. For the same reason, you can't easily scale this platform beyond a single board, which puts your clustered system back into a single point of failure.

Strange, but terrible design decision.