[seabrookmx]

This.

In fact, it's really common to use a ZFS array on single nodes, and then create a SAN using multiple such machines by layering Ceph on top.

[louwrentius]

That's interesting, but it's layers upon layers... (RIP latency), I think. Unless it's about just bandwidth and volume, then latency is not that big of a deal.

[seabrookmx]

You don't have to use ZFS snapshots. I haven't run a system like this in production but presumably you choose ZFS because it's flexible in how you configure the arrays (as is say, LVM) and because it supports checksumming.