Hacker News new | ask | show | jobs
MinLZ: Efficient and fast Snappy/LZ4 style compressor in Go (Apache 2.0) (github.com)
18 points by klauspost 454 days ago
1 comments
klauspost 454 days ago
I just released about 2 years of work on improving compression with a fixed encoding LZ77 style compressor.

Our goal was to improve compression by combining and tweaking the best aspects of LZ4 and Snappy.

The package provides Block (up to 8MB) and Stream Compression. Both compression and decompression have amd64 assembly that provides speeds of multiple GB - typical at memory throughput limits. But even the pure Go versions outperform the alternatives.

Full specification available.

link
klauspost 454 days ago
Writeup on design and changes: https://gist.github.com/klauspost/a25b66198cdbdf7b5b224f670c...

Block and Stream Format: https://raw.githubusercontent.com/minio/minlz/refs/heads/mai...

link
ptman 453 days ago
The blog post is quite nice: https://blog.min.io/minlz-compression-algorithm/

But comparisons with zstd & lz4 would be nice.

Is someone planning on a wuffs/rust implementation with C API so that it wouldn't be Go only?

link
klauspost 453 days ago
The main reason I took the effort to write the spec was to encourage ports - and commit to the format.

I've converted a basic block encoder/decoder to C (see end of README for link), to assist anyone wanting to get started on a C implementation.

Having been 10+ years since I've done any serious C, I didn't want to assume I could write a good C API for it.

If anyone are interested in picking it up, reach out.

> comparisons with zstd & lz4

The repo has lz4 comparisons (Go version) for both blocks and streams.

I see MinLZ more like a compliment to zstd, just like lz4 - where speed matter more than ultimate size.

Here is the fully parallel speed of block decompressing with zstd or minlz - Go versions:

* Protobuf - zstd: 31,597 MB/s - MinLZ 155,804 MB/s. * HTML - zstd: 25,157 MB/s - MinLZ 82,292 MB/s. * URL List - zstd: 16,869 MB/s - MinLZ 45,521 MB/s. * GEO data - zstd: 11,837 MB/s - MinLZ 36,566 MB/s.

Of course zstd compresses to a smaller size - but for things like streams transfers or for fast readbacks from disk you probably want the fastest.

Overall, I would say that MinLZ is about 2x faster decompressing, and compresses about 2x the speed compressing at max speed. But of course with less compression.

link