Hacker News new | ask | show | jobs
by hedora 3287 days ago
This is true, there is low algorithmic novelty in lz4.

However, that's missing the point: lz4's decompressor is simultaneously simple and also the fastest thing aroud, at least at the moment.

In fact, it is so fast, it can be used to accelerate local data transfers over "slow" links like bonded 10GBit ethernet and arrays of PCIe SSD's.
2 comments
Darkenetor 3287 days ago
> lz4's decompressor is simultaneously simple and also the fastest thing aroud, at least at the moment

The proprietary LzTurbo [0] and free Lizard [1] both claim to be faster at decompression while having better compression ratios.

[0] https://github.com/powturbo/TurboBench

[1] https://github.com/inikep/lizard

link
kzrdude 3287 days ago
Doesn't that also explain why it came later? The low compression ratio is useful nowadays, with large throughputs.
link
hedora 3287 days ago
Maybe. It might just be that the tricks it plays matter a lot on newer CPUs, and older fast compressors played older tricks.

I think the ratio of CPU cycles to i/o bandwidth is what really matters. Presumably the optimal tradeoff between CPU throughout and compression ratios depends on that and varies over time.

link
TheDong 3287 days ago
The low compression ratio / fast speed trade-off has always been valuable.

For example, block-level filesystem compression (what lz4 is used for with zfs) has been valuable for filesystems essentially forever.

There's always been a need for very high throughput on the local filesystem.

link