[wkat4242]

> Lesson: Test backups continuously so you get immediate feedback when they break.

This is a very old lesson that should have been learned by now :)

But yeah the rest of the points are interesting.

FWIW I rarely use ZFS native encryption. Practically always I use it on top of cryptsetup (which is a frontend for LUKS) on Linux, and GELI on FreeBSD. It's a practice from the time ZFS didn't support encryption and these days I just keep doing what I know.

[GauntletWizard]

I really love ZFS Native encryption, but this is the big problem with it. I use ZFS Raw Sends to store my backups incrementally in a cloud I trust, but not enough to have raw access to my files. ZFS has great attributes there, theoretically - I can send delta updates of my filesystems, and the receiver never has they keys to decrypt them.

I've used this in practice for many years (2020), and aside from encountering exactly this issue (though thankfully I did have a bookmark already in place), it's worked great. I've tested restores from these snapshots fairly regularly (~ quarterly), and only once had an issue related to a migration - I moved the source from one disk to another. This can have some negative effects on encryptionroots, which I was able to solve... But I really, really wish that ZFS tooling had better answers to it, such as being able to explicitly create and break these associations.

[wkat4242]

Yeah I use different methods for that. I considered using zfs send/receive for backups, however there's one big issue with that: every time you need one or two files from the backup you need to restore the whole filesystem. There's no official way to retrieve a single file from a zfs send stream.

For backup purposes I also greatly prefer file by file encryption because one corruption will only break one file and not the whole backup.

What I do now is encrypt with encfs and store on a S3 glacier-style service.

[benlivengood]

I can only think of one situation where it's hard to retrieve a remote backup of an individual file; where ZFS native encryption is in use and the remote backup system is not trusted to load the key for the dataset.

For myself, I don't trust remote systems to always have keys loaded, but in an emergency I would feel relatively safe temporarily loading the key, mounting the snapshot read-only, and scp-ing the files out, then unmounting and unloading the key (and rebooting for good measure).

There's also a viable slow option; export the raw storage of the backup ZFS pool over the (inter)network to a trusted machine and import the pool read-only locally, load the key, mount the filesystem, and make a copy. Much slower but is practical. I've used s3backer fairly successfully as a backup method for a pool with native encryption; it takes a minute or so to import the pool and can write backup snapshots at a few MB/s, so there shouldn't be any fundamental reason iscsi or similar wouldn't work.

[GauntletWizard]

I've never had to restore a single file that's older than my local snapshots; The restores where I've needed an old subset have been a substantial enough subset that 4-5xing the data size on restore was not really an issue.

I kinda agree with your point on file-by-file encryption, but ZFS's general integrity features are such that I'm not really worried - Except about this article's specific failure mode, which is pretty easy to deal with/avoid when you know about it, but is a substantial deficiency.

[mentalpagefault]

You don't actually have to restore the entire snapshot if you just want a single file! ZFS mounts snapshots read-only in an extra hidden .zfs/snapshot directory that doesn't even show up in ls -a unless you set snapdir=visible on the dataset but you can copy files out of there.

For example, cp /path/to/dataset/mountpoint/.zfs/snapshot/<snapshot_name>/path/to/file ~/path/to/file

[binwiederhier]

ZFS encryption is much more space efficient than dmcrypt+unencrypted ZFS when combined with zstd compression. This is because it can do compress-then-encrypt instead of encrypt-then-(not-really-)compress. It is also much much faster.

Source: I work for a backup company that uses ZFS a lot.

[zielmicha]

Can you explain this in more detail? It doesn't seem true on a first glance.

If you enable compression on ZFS that runs on top of dmcrypt volume, it will naturally happen before encryption (since dmcrypt is the lower layer). It's also unclear how it could be much faster, since dmcrypt generally is bottlenecked on AES-NI computation (https://blog.cloudflare.com/speeding-up-linux-disk-encryptio...), which ZFS has to do too.

[binwiederhier]

Oh my bad. I misread your comment. You are doing ZFS on top of dmcrypt, not dmcrypt images/volumes on top of ZFS.

[wkat4242]

It was my comment and yes indeed that's what I'm doing. Zfs on top of luks

[wkat4242]

I don't use compression anyway. I don't like the way that the storage pool capacity becomes variable then.

[HumanOstrich]

I don't understand. You don't like that some things compress better than others, saving a variable amount of space?

[heavyset_go]

It also saves on wear

[heavyset_go]

Using any file system that supports compression on top of LUKS does compression before encryption

[tom_alexander]

I use native ZFS encryption because it makes it super easy to share encrypted datasets across dual-booted operating systems. AFAIK Linux does not support GELI and FreeBSD does not support LUKS. DragonflyBSD supports LUKS but then no ZFS.

Also, that way I can have Linux and FreeBSD living on the same pool, seamlessly sharing my free space, without losing the ability to use encryption. Doing both LUKS and GELI would requiring partitioning and giving each OS its own pool.

[Dylan16807]

It's hard to write a completely automated backup test that's also pretty thorough. Yeah it would have caught "completely umountable" but there's a lot of other problems that a basic script has little hope of catching.

I do manual backup checks, and so did the author, but those are going to be limited in number.