[magicalhippo]

On a semi-related note, there was recently a discussion[1] on the F3 file format, which also allows for format-aware compression by embedding the decompressor code as WASM. Though the main motivation for F3 was future compatibility, it does allow for bespoke compression algorithms.

This takes a very different approach, and wouldn't require a full WASM runtime. Though it does have the SDDL compiler and runtime, though I assume it's a lighter dependency.

[1]: https://news.ycombinator.com/item?id=45437759 F3: Open-source data file format for the future [pdf] (125 comments)

[lifthrasiir]

As someone seriously trying to develop a compressed archive format with WebAssembly, sandboxing is actually easy and that's indeed why WebAssembly was chosen. The real problem is determinism, which WebAssembly does technically support but actual implementations may vary significantly. And even when WebAssembly can be made fully deterministic, function calls made to those WebAssembly modules may still be undeterministic! I tried very hard to avoid such pitfalls in my design, and it is entirely reasonable to avoid WebAssembly due to these issues.

[bangaladore]

I'm confused why determinism is a problem here? You write an algorithm that should produce the same output for a given input. How does WASM make that not deterministic?

[lifthrasiir]

Assume that I have 120 MB of data to process. Since this is quite large, implementations may want to process them in chunks (say, 50 MB). Now those implementations would call the WebAssembly module multiple times with different arguments, and input sizes would depend on the chunk size. Even though each call is deterministic, if you vary arguments non-deterministically then you lose any benefit of determinism: any bug in the WebAssembly module will corrupt data.

[bangaladore]

But that is the case in any language and runtime? There is nothing unique about WASM here.

[lifthrasiir]

Yes and that's exactly my point. It is not enough to make the execution deterministic.

Thinking about that, you may have been confused why I said it's reasonable to avoid WebAssembly for that. I meant that a full Turing-complete execution might not be necessary if that makes it easier to ensure the correctness; OpenZL graphs are not even close to a Turing-complete language for example.

[TiredOfLife]

And no mention of zpaq that has had emedable decompressors feature for 15 years

[blank_state]

you did not read the white paper then

[snapplebobapple]

Isnt that a huge vector for viruses if exevutable code is included in the compressed archive?

[themerone]

Wasm can be sandboxed. Its a safe as visiting a website with javascript.

[orangeboats]

Can't the decompressor still produce a malicious uncompressed file?

[tlb]

Any decompressor can produce a malicious file. Just feed a malicious file to the compressor.

[orangeboats]

Yes, but currently the decompressors we use (so things like zstd, zlib, 7z) come from a mostly-verifiable source -- either you downloaded it straight from the official site, or you got it from your distro repo.

However, we are talking about an arbitrary decompressor here. The decompressor WASM is sandboxed from the outside world and it can't wreak havoc on your system, true, but nothing stops it from producing a malicious uncompressed file from a known good compressed file.

[mort96]

The format-specific decompressor is part of the compressed file. Nothing here crosses a security boundary. Either the compressed file is trustworthy and therefore decompresses into a trustworthy file, or the compressed file is not trustworthy and therefor decompresses into a non-trustworthy file.

If the compressed file is malicious, it doesn't matter whether it's malicious because it originated from a malicious uncompressed file, or is malicious because it originated from a benign uncompressed file and the transformation into a compressed file introduces the malicious parts due to the bundled custom decompressor.

[yorwba]

If the decompressor is included in the compressed file and it's malicious, the file can hardly be called known good.

[jo-m]

So, not very safe.

[snapplebobapple]

I think this is the first time a genuine technical question of mine rather than a social view has been downvoted here. Thats sad.