[ackshi]

I'm a little surprised it seems they didn't have some existing compression solution before moving to zfs. With so much repetitive text across emails I would think there would be a LOT to gain, such as from dictionaries, compressing many emails into bigger blobs, and fine-tuning compression options.

[silvestrov]

They use ZFS with zstd which likely compresses well enough.

Custom compression code can introduce bugs that can kill Fastmail's reputation of reliability.

It's better to use a well tested solution that cost a bit more.

[rob_c]

Keen to move certain tasks to ZFS but not the ones that matter...

Frankly given emails are normally ~4kB objects I suspect the compression overheads are probably not that worth it unless it's for attachments only. Not attacking ZFS it's compression and checksumming are among best in class, but the compression would work better if it weren't limited to small files. Here ZFS has made a lot of wins I've not had a problem with many files on ZFS due to L1/L2 ARC but the cost is metadata ops can be painful on many small files.

The evidence they IOPS limited it that they went for SSD or better when they could store the same capacity on rust for much cheaper now.

Yeah I think moving the compression or file access up to abstract what is being written to disk ala protonmail (I don't like their offerings, but like their tech) means you can have compression over 4MB not 4kB blocks which matters when you recall data from disks for , I don't know... Backups or search?

also remember RAID!=backups ;)