[jbosh]

I love it. So much in computers is trade offs and this was a fun read exploring it.

It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?

[userbinator]

I also wonder how brotli/lzma would compare here.

Far better, just like anything else based on arithmetic coding. The main distinction here is that the output can still be decompressed with a standard Inflate implementation.

[atiedebee]

Except that brotli uses Huffman coding. It's main claim to fame is using higher order statistics to select a Huffman table and its built-in dictionary.

This class of compression programs sees larger differences due to the way the data is modelled instead of the specific entropy coder used.

[a_t48]

zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themselves.

[loeg]

To be a little pedantic, the usual zstd levels are positive integers (1-22 default 3). The negative integers denote "fast" modes with worse compression (there are only a few of these).

[edflsafoiewq]

I think those are CLI options, not negative signs. Ie. you call zstd -3 for compression level 3.

[a_t48]

Whoops! You're right, and it's too late to edit.

[chickenbig]

Even compression level 1 or 2 is pretty good.

I once used https://github.com/google/riegeli and a low zstd compression level to store large quantities of protobuf data in an efficient manner (in terms of CPU, RAM and streaming to disk). Shame Riegeli is not well known, not well documented and does not have many tests.

[Zenst]

Process-intensive, but higher compression has clear strategic value. Distant satellites such as Voyager, where bandwidth is severely limited, could transmit more data using such capabilities. Equally, for long-term archival storage, improved compression would allow far greater volumes of data to be preserved on durable, life-long media formats.

[XorNot]

Distant space probes are power constrained though.

It's entirely possible the degradation of their RTG power sources would be more expensive doing the compression then just sending the data as is.

[lstodd]

RTG degrade no matter what you do with the resulting heat. It doesn't matter if you compress stuff or just let them cpus idle be.

[XorNot]

That's the point: you're going to spend a lot more time compressing when you could've just been sending data.

And you're eating into a limited overall power and weight budget to do rather then say, run the science on the probe.