[tombert]

Nope!

I have one enclosure plugged into a USB 3.0 line, another plugged into a "super speed" line, and one plugged into a Thunderbolt line (shared with my 10GbE Thunderbolt card with a 2x20 gigabit splitter).

This was deliberate, each is actually on a separate USB controller. Assuming I'm bottlenecked by the slowest, I'm limited to 5 gigabits per RAID, but for spinners that's really not that bad.

ETA: It's just a soft RAID with ZFS, I set up each 8-bay enclosure with its own RAIDZ2, and then glued all three together into one big pool mounted at `/tank`. I had to do a bit of systemd chicanery with NixOS to make sure it mounted after the USB stuff started, but that was like ten lines of config I do exactly once, so it wasn't a big deal.

[YurgenJurgensen]

288TB spread over 24 drives on soft RAIDZ2 over USB?! You did check the projected rebuild time in the event of a disc failure, right?

[tombert]

Didn't have to do the projection, I've had to replace a drive that broke. It took about 20 hours.

ETA: It would certainly take longer with more data. I've not gotten anywhere close to the 288TB

[riobard]

Have you researched the USB-SATA bridge chips in the enclosures? Reliability of those chips/drivers on Linux used to be very questionable a few years ago when I looked around. Not sure if the situation has improved recently given the popularity of NAS devices.

[tombert]

It seems to work, and has been running for years without issue. `zpool scrubs` generally come back without issue.

[riobard]

Sounds great! Do you mind share the model of the enclosures and what bridge chip is used inside?

[Dylan16807]

From my research a couple years ago it seemed like most issues involved feeding a bridge into a port multiplier, so I got a multi drive enclosure with no multipliers. I've had no problems so far even with a disk dying in it.

Though even flaky adapters just tend to lock up, I think.

[riobard]

Ah yes! Port multiplier is usually the source of most evils (after flaky bridge chips). Unfortunately enclosure makers seldom reveal the internal topology and usually only test again Windows, and Linux kernel has a long blacklist of bad bridge chips…

[schlauerfox]

that's a benefit of zfs, it doesn't trust the drives actually wrote the data to the drives, the so called RAID write hole, since most RAID doesn't actually do that checking and drives don't have the per block checksums in a long time. It checksums to ensure.

[riobard]

The issue with flaky bridge chips wasn't usually about data integrity——it works fine most of the time, i.e. data written got read back correctly.

But often after extensive use, `dmesg` would complain about problems when talking to the drives, e.g. drive not responding or other strange error messages (forgot the exact text but very irritating and google-fu didn't help). There were also problems with SMART commands passthrough and drive power management e.g. sleep/standby adjustment which wasn't reliable when talking via bridge chips.

I use only disks directly connected to SATA controllers afterwards and no such issues ever happened again.

[jimis]

So each enclosure hosts its own RAIDZ2. Have you tested if it can survive loss of USB connectivity? It can happen because of cable damage or movement, and also because of any failure in the enclosure's electronics.