Honestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.
For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me.
I finally moved to btrfs earlier this year, and so far I'm glad I did.
I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Tailscale.
Replication is with rsync and borg, not snapshots.
Yes, it's more painful to replace disks after a failure, but once you get a bit used to it, it's really no big deal.
On my main workstation/laptop the dedupe and compression work much better than my experience with ZFS.
Isn't raid5 on raid5 perpetually broken and unsafe since inception?
Also a complicating factor with kernel upgrades is that while zfs release notes clearly delineate what kernel versions are supported that information doesn't appear to be meaningfully encoded in package metadata so if you use new enough kernels compared to the version of zfs for your distro it is possible to front run support. For instance 2.2.2 supports up to 6.6 but you could very well for instance install 6.7 and it might not work.
The somewhat broken thing is not encoding known data like which kernel is supported to automatically do the right thing in the packaging system not the filesystem. The lazy fix is to just manually handle kernel updates. The lazier one is to grab the release notes and update if latest kernel is <= supported.
The packaging was a major reason for switching to btrfs. I run sudo dnf upgrade and that's it, my system is upgraded. Zero issues ever. With ZFS I had to pin to older kernel versions, not to mention a bunch of manual steps and cleanup after any major version upgrade (every 6-12 months).
Re btrfs raid 5/6, yes everyone knows about this, and this is why I have it only on my backup system. My primary data which holds 15TB of family photos and videos is on raid 1. The offsite is there only for the time my house burns down.[1]
[1] I watched my neighbor's house go up in flames 2 years ago, and it finally got me going on setting up remote backups. The fire spread to 3 other houses, and everything happened very, very quickly. No one got hurt, but multiple families got displaced for more than a year. Besides having backups , it's also a good reminder to have adequate insurance. One neighbor did not.
I wonder if a metapackage that always depends on kernel <= supported would resolve the issue by ensuring you don't need to pin a specific version manually.
That's the way Void Linux does it. That ensures that the default kernel series always works with ZFS and NVidia modules. If you want to go off-reservation, you can do so but you're on your own, then.
The problem is that RHEL, for example, loves backporting breaking changes, so you can't know a priori that RHEL's "2.6.18" or whatever is going to keep working, and otherwise you need to push a new metapackage every time they ship an update.
Yeah, of course you should be doing that. I was just trying to say that people shouldn't pretend like having snapshot-based, filesystem-level replication alone (which can be a very efficient way of replicating data changes) is a good backup strategy.
If you don't rebuild the zfs volume on the backup computer aren't you left with a) unlimited incremental backups, or b) doing a full backup every x days?
I use LUKS / dm-crypt on entire drive partitions. Did that with ZFS too, and by the time ZFS got this feature I was already planning my migration to btrfs.
I like encrypting with LUKS, because I can have a drive configured with mine and my wife's password. Either one of us can take that drive, plug it in, and Gnome will put a nice graphical prompt asking for a password, then decrypt the drive and mount the filesystem.
If I get hit by a bus, LUKS makes it much easier for my family to get access to important data without having to have that data sit around somewhere in plaintext.
I finally moved to btrfs earlier this year, and so far I'm glad I did.
I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Tailscale.
Replication is with rsync and borg, not snapshots.
Yes, it's more painful to replace disks after a failure, but once you get a bit used to it, it's really no big deal.
On my main workstation/laptop the dedupe and compression work much better than my experience with ZFS.
[1] https://news.ycombinator.com/item?id=16797644