I've been running encrypted ZFS on 20.04 on my main workstation since it came out and it's worked great. Wrote up details here, it's a slight hack for encryption, no hack if you don't want crypto. https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-enc...
Native ZFS encryption make pool management easier. Less bookkeeping to import/export a pool. And the file system can make better decisions about performance and recovery.
I wanted to say "as someone who tends to follow the unix philosophy" and realized the irony of saying that regarding ZFS...
That said, I generally agree with you in that do one thing and do it well is a laudable design goal. However, I also am very excited about encrypted ZFS for one main reason: backups.
Okay two. Snapshots and backups!
ZFS is absolutely amazing to use as a home NAS that does daily (or more) snapshots and then nightly differential syncs to a second location. In the past I had to run all my own infrastructure to do this, as the data was in the clear.
Now my ZFS nerd friend and I can simply swap backup space and have "zero knowledge" of the others' files, while retaining the amazing features of ZFS snapshots+zfs send/receive.
This also tickles the "create an encrypted ZFS backups as a service" service itch for me, but then I realize I'd be creating it for all 13 potential users of the service. That said, I'm sure rsync.net will offer this functionality shortly - which would make them a viable backup target for me.
Regarding "unix philosophy" and ZFS... it's actually very compliant with it, despite uninformed claims from early public release about "blatant layering violations".
It's just that majority of users never have reason to see more than tiny signs of the layers hidden behind (mostly) 2 command line tools, and for various reasons those layers are compiled into one one module.
But the clean layered design is how LustreZFS happened :)
I also recall someone working years ago on a way to push snapshots to S3 or similar, but I never heard if that idea got off the ground (downside is of course the snapshots need to be recovered before they can be mounted, but the dollar cost would be rock bottom).
What would be more interesting is a backup application for Desktop Linux that assumes a ZFS root; all the problems that plague Desktop applications (that seem to keep them in eternal beta or wither away) disappear. It needs to switch on and push snapshots. It needs allow the remote file system to be mounted (to browse the snapshots for selective recovery). It needs a a disaster recovery process to recover an entire system from a remote snapshot.
In my case, because it's what Ubuntu "supports" for bootable root crypto ZFS, and I wanted to try it.
I've run ZFS on top of LUKS for my backup storage servers for probably over a decade now, and it works fine. But it wasn't really an option for my workstation.
That said, I'm not really sure what benefit I'm gaining from ZFS on my desktop. I've got that snapshots which are definitely nice. I've used it a couple times to go back in time. In theory I can go back if I wedge the system through package installs or an OS upgrade, but I've not done that (yet). It does slow down package installs because of taking snapshots, but that's ok generally.
If you're using ZFS, you're already okay with some level of ignoring that; ZFS is a inherently huge layering violation. It's a filesystem and a volume manager with encryption and compression, its own user access system (zfs allow), and its own NFS implementation.
It's a very cleanly layered system, it just doesn't bother end user with details (as implementor, you can play with them, thus LustreZFS): there's separate SPA (block), DMU (OSD) and ZPL (FS) & ZVOL (emulated block device) layers.
Compression and encryption are integrated at DMU level because that's a logical place for them.
I suppose it depends which layer you look at; as a humble sysadmin/user, it certainly behaves like a monolithic system, as opposed to having separate things to configure and invoke for LVM/LUKS/filesystem (which is, after all, the starting topic for this thread). Having only semi-recently seen an internal architecture diagram, I was indeed pleased to see that it's internally a composition of parts that do one thing well, but that's not really user-visible.
Interesting; it'd been claimed to me before that ZFS had its own NFS server (or I guess the OpenSolaris NFS server) included but that nobody used it because it was old/buggy. A quick glance at https://github.com/openzfs/openzfs/ (the old archived version based on illumos, if I read correctly) implies that this might have been true at one point, but indeed https://github.com/openzfs/zfs doesn't seem to do its own NFS so it's not true now if it ever was. Thanks for correcting my understanding.
That sounds about as modular as systemd: it's in theory modular, but it very much throws away the existing boundaries and has yet to see its new modules be adopted by external folks (if that were even possible, depending on interface stability).
I believe that the modularity will only proof itself when external (as in, from unrelated people) projects becomes established and we see how well the original project maintains compatibility.
(I must say I wonder how big the intersection of people-who-like-ZFS and people-who-like-systemd is; they seemed to originate from very different cliques but there's no reason people who like one would dislike the other…)