[mfld]

    Using larger-than-default window sizes has the drawback of requiring that the same --long=xx argument be passed during decompression reducing compatibility somewhat.

Interesting. Any idea why this can't be stored in the metadata of the compressed file?

[lifthrasiir]

It is stored in the metadata [1], but anything larger than 8 MiB is not guaranteed to be supported. So there has to be an out-of-band agreement between compressor and decompressor.

[1] https://datatracker.ietf.org/doc/html/rfc8878#name-window-de...

[mfld]

Thanks! So the --long essentially signals the decoder "I am willing to accept the potentially large memory requirements implied by the given window size"

[pbronez]

Seems useful for games marketplaces like Steam and Xbox. You control the CDN and client, so you can use tricky but effective compression settings all day long.

[chronogram]

For internal use like that you can also use the library feature. The downside of using long=31 is increased memory usage, which might not be desirable for customer facing applications like Steam.

[nolist_policy]

It uses more memory (up to +2gb) during decompression as well -> potential DoS.

[Aachen]

Sending a .zip filled with all zeroes, so it compresses extremely well, is a well-known DoS historically (zip bomb, making the server run out of space in trying to read the archive)

You always need resource limits when dealing with untrusted data. RAM is one of the obvious ones. They could introduce a memory limit parameter; require passing --long with a value equal to or greater than what the stream requires to successfully decompress; require seeking support for the input stream so they can look back that way (TMTO); fall back to using temp files; or interactively prompt the user if there's a terminal attached. Lots of options, each with pros and cons of course, that would all allow a scenario where the required information for the decoder is stored in the compressed data file

[dzaima]

The decompressed output needn't be in-memory (or even on-disk; it could be directly streamed to analysis) all at the same time, at which point resource limits aren't a problem at all. And I believe --long already is a "grater than or equal to" value, and should also be effectively a memory limit (or pretty close to one at least).

Seeking back in the input might theoretically work, but I feel like that could easily get very bad (aka exponential runtime); never mind needing actual seeking.