Hacker News new | ask | show | jobs
by yjftsjthsd-h 2326 days ago
That's part of the benefit of zstd, as I understand it; near-xz compression ratios but much faster.
2 comments

Compression is only mildly faster or on par with xz, but decompression is (at similar ratios) vastly faster. Which really helps consumers of compressed blobs.
Yeah, that's a useful distinction. So it's excellent for ex. packages (hence https://www.archlinux.org/news/now-using-zstandard-instead-o...), but iffy for write-once-read-hopefully-never backups. (Although I've heard it suggested that this might flip again the moment you start test-restoring backups regularly)
Agreed. I think I'd take zstd over xz for WORN backups anyway, just because it's pretty reliable at detecting stream corruption. (Then again, I suggest generating par2 or other FEC against your compressed backups so that's not a problem.)
Zstd has a 32-bit checksum over the uncompressed data, which is enabled by default on the CLI.
(I know you know this, as one of the principals; this is for other readers.)

And importantly, that 32-bit checksum is a pretty good checksum; a truncated XXH64(): https://tools.ietf.org/html/rfc8478#section-3 That's about as high-quality as you can expect from 32 bits. It's not, say, Fletcher-32.

Two good options here: Brotli gets within 0.6 % of the density of lzma, zstd within 5 %. Brotli is 5x faster than lzma and zstd 9x. I'm somewhat surprised that some people migrated from lzma to zstd, without considering brotli.