[wahern]

I built a benchmark harness for exploring compression of Lua script source files embedded in a Lua app binary as individual static constant C arrays. I settled on gzip, specifically deflate/inflate. Deflate does as well or better for small source files compared to xz, bzip2, and zstd.[0] More importantly, the inflate algorithm is tiny[1]; embedding the zstd decompressor blew up the binary.

I also explored building precomputed dictionaries, which means you can easily embed the files individually (C source file inclusion and runtime loading through the Lua C API remains relatively straight-forward compared to gymnastics of parsing and transforming preprocessed files) while getting similar compression ratios as when compressing an enormous file (e.g. a concatenation of all the source files). This is trivial with the zstd reference utility. It's also simple for deflate, though you have to roll your own dictionary builder by hacking the zlib implementation, or just writing it from scratch[2]. But I never bothered implementing it beyond the benchmark harness. I probably would have stuck with deflate rather than switching to zstd just because the compiled inflate implementation is so small.

[0] This is because the greatest advantage of the alternatives over deflate is the larger dictionary size they build up; deflate has a very small, upper bound. But for short inputs this advantage is diminished.

[1] https://github.com/madler/zlib/blob/develop/contrib/puff/puf...

[2] https://blog.cloudflare.com/improving-compression-with-prese...

[purplesyringa]

> small source files

How small are we talking? We've had great success with bzip-style compression on ~400 KB (about 20% better than gzip), but I don't know if it scales well. It's also relatively fast to decode (something like 2x slower than DEFLATE, IIRC).

I was also considering other approaches, specifically GLZA (https://encode.su/threads/2427-GLZA) looks promising. I think it should be well-suited for code due to its design, and it seems to produce better results than bzip on LTCB (https://mattmahoney.net/dc/text.html), and with a faster decompression time.

[wahern]

Very small, less than a few kilobytes, and many less than 1KB. I was compressing and embedding Lua source files individually. There were smarter ways to do it if I cared about overall size, but doing it per file kept the build simple. I spent way more time on the compression test harness, which was more about satisfying my curiosity and penchant for diversion than anything :) I did the tests after the actual embedding work.