|
|
|
|
|
by lifthrasiir
636 days ago
|
|
> I don't see a reason why uploaded files couldn't be (re)compressed on the server. The reason why would be of course the computational expense and latency in the package availability, which will vastly limit the algorithm choice. LZMA is probably still okay under this constraint (its compression speed is in the order of 1 MB/s for the slowest setting), but the gain from using LZMA instead of Zstandard is not that big anyway. I presume that the vast majority of big wheel files are for compiled and bundled shared libraries. (Plain Python source files are much easier to compress.) You can estimate the potential gain from looking at various compression benchmarks, such as the `ooffice` file for the Silesia benchmark [1], and the difference is not that big: 2426 KB for 7zip (which uses LZMA2 with BCJ filter) and 2618 KB for zstd 0.6.0---less than 10% difference. And I believe, from my experience, that BCJ is responsible for most of that difference because x86 is fairly difficult to compress without some rearrangement. The filter is much faster than compression and at least as fast as decompression (>100 MB/s), so there is not much reason to use LZMA when you can instead do the filtering yourself. [1] https://mattmahoney.net/dc/silesia.html |
|