[ilyt]

Throw ZFS away, put X drives, make RAID10+LVM with X-1 drives (linux supports odd numbers in RAID10), never think about it again. It's simple to setup, simple to debug, and you don't need ZFS expert for something as simple as disk replacement. In cases like what happened there is --write-mostly option that will tell linux raid to prefer other disks for reads so yo can see whether unloading the drive changes anything. Maybe RAID6 if you're not screaming for performance but want some more space.

Focus your efforts on making robust backups instead. You don't want to be that only guy in org who knows how to do ZFS things when it breaks.

We're running few racks of servers, ZFS is delegated to big boxes of spinning rust where its benefits (deduplication/compression) are used well, but on a bunch of SSDs it is just overkill.