[mr__y]

This would come with a cost of longer compression times - either multiple attempts with random shuffling or pre-compression file ordering optimization process. For resources that are compressed once and then distributed and decompressed multiple times this would be quite interesting solution

[m463]

I got curious and looked it up. On this page [1] it looks like uncompressing with gzip vs bzip2 vs xz is:

          gzip    bzip2   xz
  1       6.771   24.23   13.251
  2       6.581   24.101  12.407
  3       6.39    23.955  11.975
  4       6.313   24.204  11.801
  5       6.153   24.513  11.08
  6       6.078   24.768  10.911
  7       6.057   23.199  10.781
  8       6.033   25.426  10.676
  9       6.026   23.486  10.623

so gzip has the fastest decompression.

That said, xz is in the ballpark and can be significantly smaller.

[1] https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...

[JoshTriplett]

zstd is faster and smaller. If you can choose the format, zstd beats deflate across the board, on every front except for compatibility with things that only understand deflate.

Also, if you need to use deflate for compatibility, use https://github.com/zlib-ng/zlib-ng , which is substantially faster than either zlib or gzip.

[m463]

I've never heard of zstd (thanks!), but it seems it's right under my nose.

"Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager, and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 1300% faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used."

https://en.wikipedia.org/wiki/Zstandard

[dependenttypes]

Is it though? In https://quixdb.github.io/squash-benchmark/unstable/ deflate beats zstd at least in some tests.

[joseluisq]

Yeah, but zstd beats deflate in others. For example the latest MySQL 8 https://dev.mysql.com/worklog/task/?id=13442

[JoshTriplett]

It's universally faster if you compress threaded, and otherwise it's generally still a win.

[joseluisq]

Yeah, and it is IETF standard since 2018 https://tools.ietf.org/html/rfc8478

[vardump]

I wonder how zstd would compare.