[nunobrito]

Well, well. Kind of surprised to see this really good tool that should have been made available a longer time ago since the approach is quite sound.

When the data container is understood, the deduplication is far more efficient because now it is targeted.

Licensed as BSD-3-Clause, solid C++ implementation, well documented.

Will be looking forward to see new developments as more file formats are contributed.

[mappu]

Specialization for file formats is not novel (e.g. 7-Zip uses BCJ2 prefiltering to convert x86 opcodes from absolute to relative JMP instructions), nor is embedding specialized decoder bytecode in the archive (e.g. ZPAQ did this and won a lot of Matt Mahoney's benchmarks) but i think OpenZL's execution here, along with the data description and training system, is really fantastic.

[nunobrito]

Thanks, I've enjoyed reading more about ZPAQ but their main focus seems to be versioning (which is quite a useful feature too, will try it later) but they don't include specialized compression per context.

Like you mention, the expandability is quite something. In a few years we might see a very capable compressor.