[hks12]

It's much easier to grow if you use stripe of mirrors instead of raidz. You waste more space, but gain more flexibility and performance. More details here: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...

For example, you can start with 1 mirrored vdev, then add another mirrored vdev. You can upgrade vdevs separately, so you'll need to replace just 2 disks to grow your pool.

The only thing you have to keep in mind is that data is striped across vdevs only when written, so if you add another vdev, you won't get performance gains for data which was written just to one vdev.

[tscs37]

It's also important to keep in mind that a stripe of mirrors, aka RAID10, has a higher failure probability compared to RAIDZ2 or higher on most common disk setups by several orders of magnitude.

[rleigh]

You also have to factor in the speed of a resilver and account for the chance of a failure during this at risk window.

A mirror can resilver much faster, and it doesn't significantly affect the vdev performance while doing so. A RAIDZ resilver after a disc replacement can take a significant amount of time, and degrade performance seriously as it thrashes every single disc in the vdev.

Allan Jude and Michael Lucas' books on ZFS have tables describing the tradeoffs of the different possible vdev layouts, and they are worth a read for anyone setting up ZFS storage.

[tscs37]

The danger of loosing data on a RAID10 resilver is much higher than people expect. A 6x8TB RAID10 has a 47.3% chance of encountering a URE during a rebuild, which means data loss.

A 6x8TB RAID6 has a 0.0002% chance of a URE.

(Assuming URE rate of 10^-14, in reality this rate is lower)