[bensyverson]

The risk of z1 is that if you get a read error during resilvering, that data is permanently corrupted. The odds of this happening go way up the larger your individual drives are. This is why I chose RAIDZ2 for my NAS. I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

[Fr0styMatt88]

Or like me where you get two SSDs in your pool that happen to likely be from the same batch and both decide to permanently disappear at roughly the same time :(

[bell-cot]

> The odds of this happening go way up the larger your individual drives are.

Technically true.

Practically? There are several other major factors - such as the quality of your drives, whether you periodically run zfs scrubs and/or disk read tests, and how long a resilver takes to complete.

Behind those are the eternal duo - size of budget and cost of failure.

[Dylan16807]

> The odds of this happening go way up the larger your individual drives are.

This is a common claim, but honestly, citation needed.

We can't just apply bit error rates from the datasheet that haven't been updated in 15 years. I'm sure a 2-day rebuild is a little more risky than a 4-hour rebuild, but I'm not convinced it's by all that much. Especially if you had a monthly scrub going to prevent disk rot and disks secretly getting super fragile.

> I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.

And how many times did you have a read error on one of your other disks during a resilver?

[bell-cot]

>> The odds of this happening go way up the larger your individual drives are.

> This is a common claim, but honestly, citation needed.

FWIW, here's the quick & dirty argument for that:

Assume:

- We're working within a given generation of disk technology, with comparable drive and interface tech.

- The drives' I/O speed is relatively constant, not scaling with the drive capacity.

- The chance of any given bit (of data on a disk) going bad, in a given time interval, is relatively constant.

- The drives, regardless of size, will be some fixed-ish percentage full. (If nothing else, the boss won't sign off on overspending for unneeded capacity.)

From there - the time needed for resilvering scales linearly with the drive size. As does the number of non-redundant bits that could go bad in any given second of the resilvering. Linear time linear - so the chance of data loss during resilvering rises with the square of the disk size.

[Dylan16807]

> - The chance of any given bit (of data on a disk) going bad, in a given time interval, is relatively constant.

We can't just assume this, though. Especially for data that's being checked regularly, because the drive can see how many pre-ECC bits have gone bad and rewrite iffy sectors.

And even if it's true, it's normal to go years without errors. If you have a 15% chance of getting an error each year, and your degraded array has 4 drives without redundancy, then the chance of getting an error in 4 hours is .02% and your chance of getting an error in 48 hours is .3%. Having that risk show up once every few years is pretty negligible compared to everything else that could go wrong with your array.

On top of that, if you don't have a hot spare and it takes 5 days to get a new drive, the difference in resilvering speeds is now 5.2 days versus 7 days and that barely matters.

The main situation where you have to worry about extra drives failing is when you think the resilvering process will greatly accelerate drive deaths. And that depends very much on the specific numbers you have in mind.

[bensyverson]

> This is a common claim, but honestly, citation needed.

This is based on the data I had available 10 years ago when I set up this array… If you have data that points to drives having fewer read errors per GB in recent years, I'd love to see that citation.

> And how many times did you have a read error on one of your other disks during a resilver?

I don't know, because I have RAIDZ2, so that read error would probably be corrected without it being reported to me.

[Dylan16807]

> This is based on the data I had available 10 years ago when I set up this array

Data you collected or data you found? Any idea where it was?

I've only seen oversimplified math, not real data, and different versions of the math make different assumptions.

> I don't know, because I have RAIDZ2, so that read error would probably be corrected without it being reported to me.

Zpool status gives you a per-disk error count. I kind of assumed someone doing a resilver would use that command.

[bensyverson]

> Data you collected or data you found? Any idea where it was?

I think it was a lengthy blog post citing manufacturer data, found via the FreeNAS forums. The added safety of RAIDZ2 was a good tradeoff. Someone else on this thread reports they lost two SSDs in rapid succession, which I hear happens sometimes with drives from the same batch. In any event, I felt no need to launch an independent investigation of the math and underlying assumptions.

> Zpool status gives you a per-disk error count. I kind of assumed someone doing a resilver would use that command.

Yes, I did indeed use zpool to do the resilver… I don't recall if there were any read errors on other drives at the time of those failures, but probably not, as the scrub probably would have flagged it. They've all been replaced with higher capacity drives, so at the moment I see "No known data errors"

[Dylan16807]

Okay, interesting. Also to be clear I do think Z2 makes sense in a lot of situations, but I suspect that it's not very dependant on drive size like a lot of people think. If I wanted high reliability I'd use Z2 even with small drives.

Also to reply to your other comment: Personally one extra drive is a small price to pay for the peace of mind.

I achieve peace of mind via backups. For each copy, once I get the AFR below 2% I'm not too fussed about pushing it lower.