[krondor]

I'm going to disagree with you re: compression. Compression at the FS layer brings a lot of benefits, and generally most workloads are IO constrained not CPU constrained. Compression at the FS improves IO at the expense of CPU.

Hadoop is a completely different workload, and maybe not something for ZFS or btrfs. Our Hadoop nodes are not raid, just JBOD ext4 disks. We have been conidering btrfs with nodatacow mount option and lz4 compression, however. We haven't decided if it's better to compress within Hadoop or at the fs layer yet. I would be curious on your findings.

[jamesblonde]

In Hadoop, people are mostly using formats like Parquet, Orc, an if not, compression libs like lzo or snappy. If you believe the Berkeley people (I don't, but the sheeple do), most Spark workloads are CPU bound not IO bound. But irrespective of that, if most of your data is in a columnar data storage format, there's no gain (only cost) in having your FS also try and compress it. JBOD is considered best practice for Hadoop. That's why we're looking at RAID0 and RAID5 - we're researchers :) Actually, MapR recommend using 3 disks in RAID0 as volumes.

[ryao]

btrfs does not support lz4 compression. Unless you meant lzo, which performs terribly on incompressible data, you will want to use ZFS for lz4.

Also, nodatacow is a hack. If you take a snapshot on btrfs with nodatacow, it must use CoW on each thing in the snapshot that is overwritten. Until then, what ever horrible performance nodatacow prevented will manifest temporarily. ZFS is designed to make things asynchronous as much as possible (with the exception of partial record writes to unaccredited records, which needs to be fixed), so it lacks an equivalent to nodatacow and does not need it.

[krondor]

Yes, apologies, I did mean lzo. zlib seemed perhaps too much on the CPU side of the IO/CPU calculation. I am excited to see how btrfs snappy and lz4 support compares when they are added, however.

To the point of lzo performance though, btrfs is a tad smart with compression, it tries to compress an initial 128KiB and if the compressed segment is not smaller than the uncompressed it adds it to a list of no compress files, and will not try compression on that file again (unless of course you force it).

This was for our Hadoop use case, comparing to ext4, so nodatacow would work because we have no desire of snapshots in that environment. It still seems like we're better of compressing within the Hadoop framework (as jamesblonde is doing) and sticking with ext4 jbod, for now at least.

[ryao]

Btrfs will likely never add support for lz4 or snappy:

https://btrfs.wiki.kernel.org/index.php/Compression#Are_ther...

There are links there to mailing list emails explaining the reasoning behind that. The reasoning behind ZFS adopting LZ4 can be found here:

http://wiki.illumos.org/display/illumos/LZ4+Compression

Contrary to what the btrfs developers claimed about LZ4 versus LZJB in ZFS, LZ4's compression performance on incompressible data alone would have been enough to adopt it had ZFS already had LZO support. LZ4 also has the benefit of extremely quick decompression speeds. It also has the peculiar property where running LZ4 repeatedly on low entropy files outperforms "superior" compression algorithms such as gzip. Someone on the LZ4 mailing list discovered this when compressing log files. He compressed a 3.5GB log file into a ~750KB file by running LZ4HC 5 times. Running it twice yielded a 9.5MB file with regular LZ4 compression and a 2MB file with LZ4HC compression. He was able to compress it to ~750KB after running LZ4HC roughly 5 times.

https://groups.google.com/forum/#!topic/lz4c/DcN5SgFywwk

As for btrfs being smart with lzo by compressing only the first 128KB as a heuristic, LZ4 uses a hash table for that and is able to give up much faster. I would expect to see LZ4 significantly outperform LZO. The following site has numbers that appear to confirm that:

https://quixdb.github.io/squash-benchmark/

On a JPEG on their Intel® Core i7-2630QM, LZ4 level 1 compression runs at 608.24MB/sec while LZO level3 compression runs at 68.13MB/sec. Also of possible interest is that Snappy compresses at 559MB/s here. There are a couple caveats though. While I picked the correct variant of LZ4 for ZFS (and also the Linux kernel), I assumed that btrfs is using the default compression level on LZO like ZFS does on LZ4 and knew nothing about the different revisions measured, so I took the best reported number for any of them at the default compression level. That happened to be lzo1b. I also assumed that JPEGs are incompressible, which the data strongly supports. There are a several exceptions, but LZO, Snappy and LZ4 all consider the file to be incompressible.

As for the question of whether to compress in hadoop or in the filesystem, the properties of LZ4 mean that you can do both. If your data is incompressible, LZ4 will give up very quickly (both times). If your data is incompressible after 1 round of LZ4 compression, then the LZ4 compression in ZFS will give up quickly. If it is compressible by two rounds of LZ4, then both will run and you will use less storage space because of it.

[krondor]

Thank you, this was a great reply. Reading the discussions have been interesting.