[d33]

I worry that 7-Zip is going to lose relevance because lack of zstd support. zlib's performance is intolerable for large files and zlib-ng's SIMD implementation only helps here a bit. Which is a shame, because 7-Zip is a pretty amazing container format, especially with its encryption and file splitting capabilities.

[dikei]

I use ZSTD a ton in my programming work where efficiency matters.

But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

[cesarb]

> Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.

You can use ZSTD with ZIP files too! It's compression method 93 (see https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which is the official ZIP file specification).

Which reveals that "everyone can open a ZIP file" is a lie. Sure, everyone can open a ZIP file, as long as that file uses only a limited subset of the ZIP format features. Which is why formats which use ZIP as a base (Java JAR files, OpenDocument files, new Office files) standardize such a subset; but for general-purpose ZIP files, there's no such standard.

(I have encountered such ZIP files in the wild; "unzip" can't decompress them, though p7zip worked for these particular ZIP files.)

[throw0101d]

> You can use ZSTD with ZIP files too!

Support for which was added in 2020:

> On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[36][37]

* https://en.wikipedia.org/wiki/Zstd#Usage

So I'm not sure how widely deployed it would be.

[xxs]

Most linux distributions have zip support with zstd.

[danudey]

The `zip` command on Ubuntu is 6.0, which was released in 2009 and does not support zstd. It does support bzip2 though!

[cesarb]

> The `zip` command on Ubuntu is 6.0, which was released in 2009 and does not support zstd. It does support bzip2 though!

You probably mean the "unzip" command, which https://infozip.sourceforge.net/UnZip.html lists as 6.0 being the latest, released on 20 April 2009. Relevant to this discussion, new in that release are support for 64-bit file sizes, bzip2 compression method, and UTF-8 filenames.

The "zip" command is listed at https://infozip.sourceforge.net/Zip.html as 3.0 being the latest, released on 7 July 2008. New in that release are also support for 64-bit file sizes, bzip2 compression method, and UTF-8 filenames.

It would be great if both (or at least unzip) were updated to also support LZMA/XZ/ZSTD as compression methods, but given that there have been no new releases for over fifteen years, I'm not too hopeful.

[xxs]

i meant the "zip support" (as in zlib) with zstd command.

[dikei]

Well, only a lunatic would use ZIP with anything but DEFLATE/DEFLATE64

[redeeman]

there are A LOT of zip files using lzma in the wild. also, how about people learn to use updated software? should newer video compression technologies not be allowed in mkv/mp4.

if you cant open it, well.. then stop using 90ies winzip

[landl0rd]

No. You can't get people to use updated software. You can't get a number of people to update past windows 7. This has been and will likely remain a persistent issue, and it's sure not one you're going to fix. All it will do is limit your ability to work with people. This isn't a hill on which you should die.

[redeeman]

if they want to open certain files, they will update

[1over137]

>how about people learn to use updated software?

How about software developers learn to keep software working on old OSes and old hardware?

[tiagod]

What stops you from running updated zip/unzip on an old OS or on old hardware?

[redeeman]

what software doesnt support OSs that are in active SECURITY support?

[Am4TIfIsER0ppos]

mkv or mp4 with h264 and aac is good enough. mp3 is good enough. jpeg is good enough. zip with deflate is also good enough.

[e4m2]

"Good enough" is not good enough.

[dahrkael]

I started using winrar because winzip wouldnt fit in a floppy disk. so even in the 90s zip wasnt good enough

[homebrewer]

In the middle of San Francisco, with Silicon Valley level incomes, very possible. In the real world I still exchange files with users on rustic ADSL, where every megabyte counts. Many areas out there, in rural Mongolia or in the middle of Africa that's just got access to the internet, are even worse in that regard.

[redeeman]

h264 is not good enough for many things

[easton]

> new Office files

I know what you mean, I’m not being pedantic, but I just realized it’s been 19 years. I wonder when we’ll start calling them “Office files”.

[mauvehaus]

> I wonder when we’ll start calling them “Office files”.

Probably around the same time the save icon becomes something other than a 3 1/2" floppy disk.

[jl6]

English is evolving as a hieroglyphic language. That floppy disk icon stands a good chance of becoming simply the glyph meaning "save". The UK still uses an icon of an 1840s-era bellows camera for its speed camera road signs. The origin story will be filed away neatly and only its residual meaning will be salient.

[kevinventullo]

Nowadays I’ve noticed fewer applications have a save icon at all, relying instead on auto-save.

[ale42]

And some only save to the cloud, whence a cloud icon with an arrow. (Not that I like that, but... that's what we get)

[guappa]

You can and I've done it… but you can't expect anything to be able to decompress it unless you wrote it yourself.

[sidewndr46]

Same thing with "WAV" files. There's at least 3 popular formats for the audio data out there.

[martinald]

More 'useful' one is webp. It has both a lossy and lossless compression algorithm, which have very different strengths and weaknesses. I think nearly every device supports reading both, but so many 'image optimization' libraries and packages don't - often just doing everything as lossy when it could be lossless (icons and what not).

[LegionMammal978]

It's similarly annoying how many websites take the existence of the lossy format as a license to recompress all WebP uploads, or sometimes other filetypes converted to WebP, even when it causes the filesize to increase. It's like we're returning to ye olden days of JPEG artifacts on every screenshot.

[danudey]

I was thinking about this with YouTube as an example. A lot of people complain about the compression on YouTube videos making things look awful, but I bet there's a reasonable number of high-end content creators out there who would run a native(-ish, probably Electron) app on their local system to do a higher-quality encoding to YouTube's specifications before uploading.

In many (most?) cases, it's possible to get better compression and higher quality if you're willing to spend the CPU cycles on it, meaning that YouTube could both reduce their encoding load and increase quality at the same time, and content creators could put out better quality videos that maintain better detail.

It would certainly take longer to upload the multiple multiple versions of everything, and definitely it would take longer to encode, but it would also ease YouTube's burden and produce a better result.

Ah well, a guy can dream.

[sidewndr46]

So apparently webp is also 'RIFF" which is the container for WAV files as well it seems. I did not know this. Also webp has its own specialized lossless algorithm. For things like icon art I generally just continue to use PNG. Is there an advantage to using Webp Losslesss?

[justin66]

> Copyright (c) 1989 - 2014, 2018, 2019, 2020, 2022

Mostly it seems nutty that, after all these years, they’re still updating the zip spec instead of moving on to a newer format.

[pornel]

The English language is awful, and we keep updating it instead of moving to a newer language.

Some things are used for interoperability, and switching to a newer incompatible thing loses all of its value.

[6SixTy]

.7z and .tar.* have existed for at least 20 years now, but you are unlikely to see a wild 7z file and .tar.* is isolated to the UNIX space

[danudey]

Tar files also have the miserable limitation of having no index; this means that to extract an individual file requires scanning through the entire archive until you find it, and then continuing to scan through the rest of the archive because a tar file can have the same file path added multiple times.

That makes them useful for transferring an entire set of files that someone will want all or none of, e.g. source code, but terrible for a set of files that someone might want to access arbitrary files from.

[justin66]

Sure, but that's not really a reason to futilely try to spooge oddball algorithms that nobody is going to adopt into the .zip standard.

[Akronymus]

So thats why rarely the customer cant open one of the zip files we send over.

[notepad0x90]

I don't know about, had a dicey situation recently where powershell's compress-archive couldn't handle archives >4GB and had to use 7zip. it is more reliable and you can ship 7za.exe or create self-extracting archives (wish those were more of a thing outside of the windows world).

[chasil]

In the realm of POSIX.2 and UNIX relatives, the closest analog would be a "shar" archive.

They are not regarded kindly.

https://en.wikipedia.org/wiki/Shar_(file_format)

[landl0rd]

I understand that security has to compromise for the real world, but a self-extracting archive is possibly one of the worst things one could use in terms of security.

[notepad0x90]

why? and why does it have to be a compromise?

You're assuming things because things are already done insecurely. You can authenticate the self-extractor as well as the extracted content. The user gets a nice message "This is a 7zip self-extracting archive sent to you by Bob containing the files below".

As an incident responder, I've seen much more of regular archives being used to social engineer users than self-extracting archives, because self-extracting is not "content executing". it is better for social engineering for users to establish trust in the payload first by having them manually open the archive. if something "weird" like self-extraction happens first, it might feel less trustworthy.

Oh and by the way, things like PyInstaller or electron apps are already self-extracting and self-executing archives. So are JAR files and android APK's.

[fsckboy]

jar files are zip files, so they don't contain "self extract" code, instead they are associated with already installed extraction code.

however, once extracted, jar files do contain executable code, and that is a security issue. the java model pays attention to security, but if code can do something, it can do something bad. if it can't do something, it's not very useful, is it.

[notepad0x90]

the windows kernel executes a self extracting 7z archive, java.exe extracts and executes .jar files. If the 7z self extractor was .net CLR bytecode it would operate very much the same as .JAR files. to your point though, the primary purpose of JAR files is not to compress and transport other files, they're supposed to be executables only. From a user's perspective, abuse potential is the main difference.

[sidewndr46]

What are you compressing with zstd? I had to do this recently and the "xz" utility still blows it away in terms of compression ratio. In terms of memory and CPU usage, zstd wins by a large margin. But in my case I only really cared about compression ratio

[vlovich123]

people tend to care about decompression speed - xz can be quite slow decompressing super compressed files whereas zstd decompression speed is largely independent of that.

People also tend to care about how much time they spend on compression for each incremental % of compression performance and zstd tends to be a Pareto frontier for that (at least for open source algorithms)

[bracketfocus]

This makes sense. A lot of end-users have internet speeds that can outpace the decompression speeds of heavily compressed files. Seems like there would be an irrational psychological aspect to it as well.

Unfortunately for the hoster, they either have to eat the cost of the added bandwidth from a larger file or have people complain about slow decompression.

[vlovich123]

Well the difference is quite a bit more manageable in practice since you’re talking about single digit space difference vs a 2-100x performance in decompression.

[sidewndr46]

I definitely agree, I basically have unlimited time and unlimited CPU for decompressing. Available memory is huge too. The gains from xz were significant enough that I went with it.

[landl0rd]

I usually see zstd on max settings outperform xz on speed and very slightly on compression (though that's a tiny difference).

[Szpadel]

in my experience using zstd --long --ultra -22 gives marginally better compression ratio than xz -9 while being significantly faster

[soruly]

I think it depends on what you're compressing. I experimented with my data full of hex text xml files. xz -6 is both faster and smaller than zstd -19 by about 10%. For my data, xz -2 and zstd -17 achieve the same compressed size but xz -2 is 3 times faster than zstd -17. I still use xz for archive because I rarely needs to decompress them.

[Szpadel]

Try combining it with --long

My use cases are usually source code, SQL dumps and log files.

Sometimes xz gave marginally better results, but difference was well below 1%

[soruly]

thanks for the tips. As my data has very low entropy, both can compress down to 3-4% of original size, but xz is a lot faster in compression.

raw size: 9612344 B

zstd --ultra -22 --long=31 => 376181 B (3.91% original, 4.088s compress, 0.013s decompress)

xz -z -9 xml => 353700 B (3.68% original, 0.729s compress, 0.032s decompress)

zstd -17 --long=31 could match the compression time of xz, but the size is bigger (405602 B, 4.22% original)

If you compare only the compressed size (not to the original size), .zst would be about 6-15% larger than .xz

[xxs]

do you have examples where xz 'blows it away', not just zstd -3?

[sidewndr46]

Here are some examples of what I was doing in one case

https://www.hydrogen18.com/blog/apk-the-strangest-format.htm...

I was running "zstd --ultra --threads=0" which I assumed was asking it for the absolute maximum

[sltkr]

I think your mistake was to use --ultra without a compression level.

I redid your experiments with rust-wasm-1.83.0-r0.apk:

                            size       perc   c.time  d.time
    uncompressed:      290072064          -        -
    gzipped original:  105255109     36.29%        -  
    bzip2 -9:          107099379     36.92%    21.1s  11.0s
    bzip3 -b511:        73539847     25.35%    28.9s  32.0s
    xz --extreme -9:    71010672     24.48%   142.0s   3.1s
    lzip -9:            70964413     24.46%   173.5s   5.3s
    zstd --ultra -22:   48288499     16.64%   155.6s   0.4s

It's pretty clear zstd blows everything else out of the water by a huge margin. And even though compressing with zstd is slightly slower than xz in this case (by less than 10%), decompression is nearly 8x as fast, and you can probably tweak the compression level to make zstd be both faster and better than xz.

[ars]

That was an impressive result, so I tried it on a huge email inbox.

    uncompressed:    1512662084
    xz --extreme -9:  508431572  12:47
    zstd --ultra -21: 508432560  12:44

(-22 ran out of memory.) So at least by me zstd was identical to xz almost to the byte and the second.

[sidewndr46]

I guess I misunderstood the man page for that option then.

[xxs]

yup, you should have tried just different -NN, and notice. I had a talk on zstd couple of years back, and one of the points was that it was better than xz across the board.

[jart]

Use the pigz command for parallel gzip. Mark Adler also has an example floating around somewhere about how to implement basically the same thing using Z_BLOCK.

[mrWiz]

My main use case for 7z is bypassing corporate filters that block ZIPs from being sent.

[starik36]

I think gmail is onto you. They blocked one of my 7z files the other day.

[mrWiz]

Thankfully our corporate IT isn't onto me yet.

[psyclobe]

zip is such a shit standard, hell there are parts of it that are still undocumented and sharing documents between system zip implementations across mac and windows sometimes fails.

[Night_Thastus]

7-zip is the de-facto tool on Windows and has been for a long time. It's more than fast and compressed enough for 99% of peoples use cases.

It's not going anywhere anytime soon.

The more likely thing to eat into its relevance is now that Windows has built-in basic support for zipping/unzipping EDIT: other formats*, which relegates 7-zip to more niche uses.

[Bender]

7-zip is the de-facto tool on Windows and has been for a long time.

Agreed. The only thing I think it has been missing is PAR support. I think they should consider incorporating one of the par2cmdline forks and porting that code to Windows as well so that it has recovery options similar to WinRAR. It's not used by everyone but that should deprecate any use cases for WinRAR in my opinion.

[malfist]

Windows has had built in zip/unzip since vista. 7zip is far superior (and the install base proves that)

[Night_Thastus]

As mentioned in another comment, zip support actually goes further back as far as '98, but only Windows 11 added support for handling other formats like RAR/7-Zip/.tar/.tar.gz/.tar.bz2/etc.

That allows it to be a default that 'just works' for most people without installing anything extra.

The vast majority of users don't care about the extra performance or functionality of a tool like 7-zip. They just need a way to open and send files and the Windows built-in tool is 'good enough' for them.

I agree that 7-zip is better, but most users simply do not care.

[landl0rd]

Windows zip is not in fact good enough. I've run into weird, buggy behavior, hanging on extract, all sorts of nonsense. I can see the argument that a universally-adopted solution is better, but that's different from windows just not working.

[Night_Thastus]

I'm not saying I would ever use it. I'm saying that for casual non-power users, it's good enough. They work with it and if it breaks once in a blue moon they don't care. They just want it to open the files they get and give them a way to send files compressed.

That is enough to bite into 7-Zip's share of users.

[iamleppert]

Windows unzip is so ungodly slow and terrible! Long live 7zip!

[sixothree]

Windows now has 7z support. I fear 7-zip will continue to lose relevancy for consumers of .7z files.

[_zagj]

7-zip, through its .7z format, also supports AES encryption. I'd argue it's probably the easiest way to encrypt individual file archives that you need to access on both Windows and Linux. I have a script I periodically run that makes an encrypted .7z archive of all of my projects, which I then upload for off-site backup. (On-site, I don't bother encrypting.)

[izzydata]

Is there something different about the built in zip context menu functionality now than before? I'm pretty sure you could convert something to a zip file since forever ago by right clicking any file.

[Night_Thastus]

It could support basic ZIP files, but only Windows 11 added support for 7-Zip (.7z), RAR (.rar), TAR, and TAR variants (like .tar.gz, .tar.bz2, etc).

That makes it 'good enough' for the vast majority of people, even if it's not as fast or fully-featured as 7-Zip.

[rf15]

Not that many people care about zstd; I would assume most 7-zip users care about the convenience of the gui.

[arp242]

It's been a long time since I used Windows, but back in the day I used 7-Zip exactly because it could open more or less $anything. That's also why we installed it on many customer computers.

On Linux bsdtar/libarchive gives a similar experience: "tar xf file" works on most things.

[devilbunny]

7-Zip is like VLC: maybe not the best, but it’s free (speech and beer) and handles almost anything you throw at it. For personal use, I don’t care much about efficient compression either computationally or in terms of storage; I just want “tar, but won’t make a 700 MB blank ISO9660 image take 700 MB”.

[tssva]

Windows 11 has shipped with bsdtar/libarchive for a few years. The gui shell support for archive files was recently changed to use libarchive which has increased the supported archive files which can be opened in the shell.

[KronisLV]

That's basically me! I really like 7-Zip because it opens most archive formats I have to work with and also the .7z format has pretty good compression for the stuff I want to store longer term.

[cm2187]

in fact this is the first time I even hear about it, and I am semi-IT litterate. The prevalence of a compression standard is about how ubiquitous it is. For that one, I would vote "not even on the radar yet".

[yapyap]

if by gui u mean the ability to right click a .zip file and unzip it just through the little window that pops up ur totally right. At least that + the unzipping progress bar is what I appreciate 7zip for

[snickerdoodle12]

That's why 7zip should support it. People care about the convenience of the GUI and we all benefit from better compression being accessible with a nice GUI.

[Beretta_Vexee]

I just hope that the recipient will be able to open the file without too much difficulty. I am willing to sacrifice a few megabytes if necessary.

[quickthrowman]

I use the right click context menu to run 7zip, why would you open a GUI?

[quietbritishjim]

That is a GUI!

[jorvi]

.. but 7-zip has a pretty terrible GUI?

Hence why PeaZip is so popular, and J-Zip used to be before it was stuffed with adware.

[sidewndr46]

If you're expecting a "mobile first" or similar GUI where most of the screen is dedicated to whitespace, basic features involves 7 or more mouse clicks and for some reason it all gets changed every ~6 months then yes the 7zip GUI is terrible.

Desktop software usability peaked sometime in the late 90s, early 2000s. There's a reason why 7zip still looks like ~2004

[wmil]

When compared to it's contemporaries the 7-zip GUI is noticeably worse. Back in 2004 WinRar and WinZip were both clearly superior.

[birksherty]

Not sure what those gui improvments in winrar is. I prefer 7zip over all those for gui too.

[jorvi]

You could have taken the 10 seconds to type in "PeaZip GUI" and seen that it is not a mobile interface and it is indeed much nicer than the 7-Zip interface.

Instead you chose to make a useless snarky comment. Be better.

[general1726]

Most people won't use that GUI, but will right click file or folder -> 7-Zip -> Add To ... and it will spit out a file without questions.

Granted Windows 11 has started doing the same for its zip and 7zip compressors.

Same trick goes for opening archives or executables (Installers) as archives.

[axus]

Let's chat about Windows 11 right-click menu. I'm pretty sure they hid all the application menu extensions to avoid worst-case performance issues.

[p_ing]

Exactly it. 3rd parties injecting their extensions harmed performance, which people turn around and blame Microsoft for.

[birksherty]

People are really praising microsoft for for all three new "features" and tracking in win 11.

[m-schuetz]

All the GUI I need is right click-> extract here or to folder. And 7zip is doing that nicely.

[Jackson__]

PeaZip is popular? It seems a lot less tested than 7zip; Last time I tried to use it, it failed to unpack an archive because the password had a quote character or something like that. Never had such crazy issues in 7zip myself.

[Gormo]

> .. but 7-zip has a pretty terrible GUI?

Since you're asking, the answer is no. 7-Zip has an efficient and elegant UI.

[tssva]

I find the PeaZip gui to be awful to use. I much prefer the 7-zip gui.

[delfinom]

I would never trust PeaZip.

The author updates code in the github repo....by drag and drop file uploads. https://github.com/peazip/PeaZip/commits/sources/

[Beretta_Vexee]

You are looking for 7-Zip Zstd: https://github.com/mcmilk/7-Zip-Zstd

I don't know what your use case is, but it seems to be quite a niche.

[zx2c4]

I was curious upon seeing this and found the thread where its inclusion was turned down: https://sourceforge.net/p/sevenzip/discussion/45797/thread/a...

[birksherty]

I use this one too. This is the best one out of all the alternative 7zip with zstd. UI is same, has all the options and faster than those for zstd in my comparisons.

[sammy2255]

https://github.com/mcmilk/7-Zip-zstd

[abhinavk]

https://github.com/M2Team/NanaZip

It includes the above patches as well as few QoL features.

[birksherty]

Tried it, compression is worse compared to 7-Zip-zstd by mcmilk for zstd, for same speed. The removal of text from toolbar icons is enough for me to never use it again. 7zip can change file associations directly with it and very easy. Feels like NanaZip is worse in QoL than 7zip.

[d33]

Thanks! Any ideas why it didn't get merged? Clearly 7-Zip has some development activity going on and so does this fork...

[Beretta_Vexee]

Working with Igor Pavlov, the creator of 7-zip, does not seem very straightforward (understatement).

[Tuldok]

7-zip's development is very cathedral. Igor Pavlov doesn't look like he accepts contributions from the public.

[m-schuetz]

Being a bit faster or efficient won't make most people switch. 7z offers great UX (convenient GUI and support for many formats) that keeps people around.

[rat9988]

If anything, the gui and ux is terrible compared to winrar.

[jccalhoun]

Since Windows 11 incorporated libarchive back in October 2023 there is less reason to use 7-zip on windows. I would be surprised if any of my friends even know what a zip file is let alone zstd.

[rs186]

If you ever try to extract an archive file of several gigabyte size with hundreds of thousands of files (I know, it's rare), the built-in one is as slow as a turtle compared to 7z.

[pjmlp]

As long as it does a better job than whatever Windows team packs into the OS, they're safe.

Even on latest Windows 11 takes minutes to do what 7-Zip does in seconds.

Goes to show how good all those leetcode interviews turn out.

[conkeisterdoor]

Glad I'm not the only one who feels this way. WinZip is a slow and bloated abomination, especially compared to 7-Zip. The right-click menu context entry for 7-Zip is very convenient and runs lightning fast. WinZip can't compete at all.

[pjmlp]

Mixing channels here, WinZip is a commercial product, unrelated to Windows 11 7-zip support, and my comment.

https://www.winzip.com

[xxs]

There are lots of 7zip alike with zstd support (it's a plugin effectively). On [corporate] Windows NanaZip would be my choice as it's available in Windows store.

on anything else - either directly zstd or tar

[marcellus23]

Why are they not adopting ztsd?

[privatelypublic]

It already has- look up nanazip