[rubenbe]

For me the killer feature of btrfs is "RAID 1 with different sized disks". For a small and cheap setup, this is perfect since a broken disk can be replaced with a bigger one and immediately (part of) the extra new disk space can be used. Other filesystems seem to only increase the size once all disks have been replaced with a bigger capacity one (last time I checked this was still the case for ZFS)

[frankjr]

Exactly. Provisioning a completely different set of disks when running out of capacity might be fine for a company but not for home office.

[e12e]

How does that work? You have two 100gb drives in raid1, 80% full, you replace one with a 200gb disk and write 50gb to the array - how is your 130gb of data protected against either drive failing?

[dspillett]

I don't know the ins-and-outs of btrfs in detail, but having dug into other systems that offer redundancy-over-uneven-device-sizes and assuming btrfs is at least similar: with your two drive example you won't be able to write another 50gb to that array.

For two devices, 1x redundancy (so 2x copies of everything) will always limit your storage to the size of the smaller device otherwise it is not possible to have two copies of everything you need to store. As soon as you add a third device of at leats 100gb (or replace the 100gb device with one at least 200gb) the other 100gb of your second device will immediately come into play.

Uneven device size support is most useful when:

♯ You have three or more devices, or plan to grow onto three or more from an initial pair.

♯ You want flexibility rwt array growth (support for uneven devices usually (but not always) comes with better support for dynamic array reshaping).

♯ You want better quick repair flexibility: if 4Tb drive fails, you can replace it with 2x2Tb if you don't have a working 4Tb unit on-hand.

♯ You want variable redundancy (support for uneven devices sometimes comes with support for variable redundancy: keeping 3+ copies of important data, or data you want to access fastest via striping of reads, 2 copies of other permanent data, and 1 copy of temporary storage, all in the same array). In this instance the “wasted” part of the 200gb drive in your example could be used for scratch data designated as not needing to be stored with redundancy.

[rubenbe]

It only works with 3+ disks. All data needs to be on two disks.

e.g. you have 3 100GB drives, total capacity in raid 1 is 150GB.

If you replace a broken one with a 200GB one, the total capacity will be increased to 200GB.

[assbuttbuttass]

My understanding is that RAID1 is just a mirror, and all disks have identical contents. Are you talking about something else?

[tuetuopay]

Traditional RAID1 will mirror whole drives, yes. BTRFS RAID1 will mirror chunks of data (iirc 1GB) on two drives. So you can have e.g. two 1TB drive and a 2TB one just fine.

[magicalhippo]

You can do this with ZFS, you just have to do it manually, ie by partitioning up the disks in say 100GB or 1TB partitions, then construct vdevs using these partitions.

You can then extend the pool by adding more such partition-based vdevs as you replace disks, just add the new partitions and add new vdevs.

So if you have a 1TB disk, a 2TB disk and a 4 TB disk, you could have mirrors (d1p1,d3p1), (d2p1,d3p2), (d2p2,d3p3) with a total of 3TB mirrored, and 1 TB available. If you swap the 1TB for a 2TB disk, partition it, replacing the old d1p1 partition with the new and resilver, and once that's done you can add the mirror (d1p2,d3p4) and get the full 4TB redundant storage.

Not a great solution though, as it requires a lot of manual work, and especially write performance will suffer because ZFS will treat the vdevs as being separate and issue IOs in parallel to them, overloading the underlying devices.

[assbuttbuttass]

Thanks, that makes sense. I think the bcachefs --replicas option does something similar

[kevincox]

Yeah, BTRFS is really not good for any sort of redundancy, not even very good for multi-disk in general.

1. The scheduler doesn't really exist. IIRC it is PID % num disks.

2. The default balancing policy is super basic. (IIRC always write to the disk with the most free space).

3. Erasure coding is still experimental.

4. Replication can only be configured at the FS level. bcachefs can configure this per-file or per-directory.

bcachefs is still early but it shows that it is serious about multi-disk. You can lump any collection of disks together and it mostly does the right thing. It tracks performance of different drives to make requests optimally and balances write to gradually even out the drives (not lasering a newly added disk).

IMHO there is really no comparison. If it wasn't for the fact that bcachefs ate my data I would be using it.

[leansensei]

That, plus offline deduplication.

[etskinner]

Bcachefs has this too