Hacker News new | ask | show | jobs
by mwcampbell 737 days ago
Interesting observations on compression. As a young programmer, I used to compress executables, and maybe some DLLs as well, with UPX without a second thought. Later I understood that executable compressors prevented the OS's memory-mapped file I/O and demand paging from working as designed, and moved to only compressing the installer and update packages (another of my misadventures as a young programmer was doing my own updater with its own package file format).

I guess the ideal solution would be if the download server offered a few compression options negotiable at download time, via Content-Transfer-Encoding or some other form of HTTP content negotiation, trading CPU time against bandwidth (the server would have to pre-compress or at least cache the compressed versions to scale), and then the download was stored as some kind of archive that could be mounted as a filesystem (this implies random access and therefore not "solid" compression). Then delta updates would be done against that filesystem image. That way, you wouldn't have the "installing" process of uncompressing and copying files. Of course, that would require platform support that we don't have on Windows and macOS. At least I can dream about desktop Linux.

1 comments

macOS actually has the best support for that. DMG files are mountable disk images and the contents can be compressed with LZMA or some Apple-specific codecs that are quite good. Opening them mounts them into the kernel and then there's random access. Even code signatures are checked JIT during page faults.

The main problem with DMGs is the poor UX, and very slow mount/verification times. Users can start the app from the DMG and it will seem to work, but be unable to update. They forget to unmount the "drive" or don't know how. The format is also undocumented and a PITA to work with as it's basically a full filesystem, which also has to be signed and notarized independently and that's super slow too. So it makes the whole build process a lot slower.

There's quite some low hanging fruit here that I might experiment with soon. I have a design in mind already.