[ultramancool]

Cool idea. Not enough people seem aware of executable packers like UPX http://upx.sourceforge.net/ though.

These are excellent tools to keep the size down when using large static binaries. By compressing the file on disk and decompressing in memory you often wind up with a smaller and sometimes even faster loading (depending on disk IO speed vs decompression speed) package. I got a static Qt binary from 4 MB down to 1.3 with upx --lzma. Very nice stuff.

[Lerc]

The downside to tools like UPX is that the executable code is actively transformed on load. This limits the ability to use shared memory for multiple concurrent executions of the same executable.

If the OS loads executables by mmap and load on page-hit, you can potentially save memory by not ever loading unused parts of an executable. a transform-on-load requires the entire program to be loaded before execution begins.

[GeorgeTirebiter]

'... never loading unused parts of an executable...' - this is one of the benefits of a paging system. Paging was (is?) the best way to keep memory requirement down. The way Multics worked was by having paged segments. http://www.osinfoblog.com/post/136/segmentation-with-paging:... The x86(-64) can support something like this, but as far as I know, no modern OS supports this feature.

[asuffield]

Linux does this, as does basically every other modern OS.

Pick a random large process on a linux host - like your web browser. cat /proc/$pid/status.

VmExe is the size of the mapped executable; VmLib is the size of all the other mapped libraries and executable pages. Add those two numbers together to find the size of all the executable code mapped into this binary.

VmRSS is the amount of physical memory that the process is currently using. You'll find that this is a lot smaller than the code mapped into the binary. That's because the kernel hasn't loaded any of that into physical memory.

[GeorgeTirebiter]

Thank you. I'm not at all sure why the 'old hands' here on HN have decided to so viciously downvote this. I didn't call anybody names, didn't violate any posting rules, did not violate etiquette, and provided a very nice link to Multics.

If there is a perceived error in what I wrote, then, like the one nice responder, explain, please.

HN is such a different community now than when I joined 1,835 days ago. It brings tears to my eyes.

[rab_oof]

Ah, and pklite pro and lzexe from the olden days for winDOwS shit once upon a time.

Though the goal should be generating less code. Link in fewer dependencies, reduce features, DRY up duplicate logic and cut LoC. Also compile with -DNDEBUG -O2 -g- and whatever LTO switches are available for whole program optimization if you're statically linking everything together. Also be sure to include static dependencies of other static dependencies like zlib (-lz), or you'll inevitably end up with missing symbol errors when compiling a final program. LTO cuts out all (most) of the shit that you don't need and attempts to optimize across translation units.

Furthermore a consideration against static linking, on most platforms, if the same shared library is already loaded, it's reused by mmaping it into a process. Not sure that duplicating code is going to reduce memory usage or the IO it takes to load from disk. Giant runtimes like Go, Ruby, Python and fucking Java shouldn't be duplicated N times... That's just wasteful. (I hate any language with an epic runtime or VM that includes the world to do anything.). Libraries should be reserved for the few redundant things that take tons of code to implement and change very little.

If anyone wants to compile a Linux system from scratch, try LFS and hackaround with static linking. It may take patches, extra flags to get what you want.

Hope Static Linux scales, because it's easier to upgrade static programs without dependency hell but the increased memory usage of duplicated code might not be so great of a tradeoff.

Another hack would be to statically compile every system program in each directory together (/sbin/, /bin/, and parts of /usr/bin, etc) into a single executable per directory that is then symlinked to itself to select which "program" to load via argv[0]. It will be one giant exe per directory, but it will be cached basically all the time and with LTO, there won't be much duplication as with N programs compiled separately. This would take a main which dispatches to other renamed original mains and renaming all symbol conflicts across all translation units.

    /bin/static
    /bin/[ -> /bin/static
    /bin/false -> /bin/static
    /bin/true -> /bin/static

(Probably want to use hard links also)

[justincormack]

Crunchgen [1] does that static compile, merge and symbol rename trick for you

[1] http://netbsd.gw.com/cgi-bin/man-cgi?crunchgen++NetBSD-curre...

[voltagex_]

I found a bug with an offline documentation reader (can't remember which) that if you pack one of the QT components (qwindows.dll) the application wouldn't load anymore. Still not sure whether to report a bug and who to report it to.

Yes UPX is cool, but I don't think it's 100% compatible.

[ultramancool]

You can usually set UPX options to fix that kind of thing. You may have stripped necessary linking information. I usually use it on statically linked things though, so no issue there.