[kaliszad]

What surprises me is, why they went with the harder to cool 1U nodes and 10 SSDs/2x100Gb NICs instead of 2U nodes with 24 SSDs/2x200 or even 400Gb NICs. They could remove the network bottleneck and save on power thanks to larger, lower speed fans and less CPU packages, possibly with more cores per socket though. Also, having a smaller number of nodes increases the blast radius but with even 34 nodes this is probably not such a problem. However, with less nodes they could have a flatter network with 4 switches or so too.

[PiratesScorn]

Blast radius is the primary factor as you say and just generally makes things like patching and HW replacements less stressful. The racks and switches already exist and are heavily utilised for other purposes so the additional physical footprint for ceph is pretty tiny :)