It would amazing to see this work on OpenWRT, I think it would fit perfectly using less resources than squashfs.
The other location would be on a Raspberry pi for scenarios where power can be cut at any time.
Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file systems that are smaller with much less CPU resources. However, it'll still need a reasonable amount of memory at run time to cache active, decompressed blocks.
Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so you need to be able to keep more than one in memory anyway. However, decompressed files are kept in the cache in the hope that further (or even concurrent) reads will access the same blocks. Taking the example from the README where over a 1000 perl binaries are being executed concurrently, that cache typically has hit rates of 99+%:
$ dwarfs perl-install.dwarfs mnt -f
23:02:42.673390 dwarfs (0.2.1)
23:02:42.676663 file system initialized [1.94ms]
23:02:49.210158 blocks created: 226
23:02:49.210189 blocks evicted: 194
23:02:49.210216 request sets merged: 123
23:02:49.210241 total requests: 50056
23:02:49.210270 active hits (fast): 1515
23:02:49.210293 active hits (slow): 833
23:02:49.210318 cache hits (fast): 47482
23:02:49.210343 cache hits (slow): 0
23:02:49.210392 fast hit rate: 97.8844%
23:02:49.210417 slow hit rate: 1.66414%
23:02:49.210441 miss rate: 0.451494%
For example, reducing the cache size from 512M (default) to 32M increases the time it takes to run 1139 binaries from 2.5 seconds to almost 40 seconds.
If you're talking about the kernel's filesystem cache, wouldn't that cache the compressed files? As far as I understand it userspace caching is necessary to cache uncompressed blocks, since the decompression is (presumably) done in userspace. I definitely could be wrong though, let me know if you're talking about a different kind of kernel caching.
Actually, I guess if DwarFS is a kernel module and decompresses blocks before they hit the kernel's filesystem cache, then the kernel cache would do it? I'm not sure how to tell from the README if DwarFS is a kernel module or not. So I guess I'm just confused and looking to learn -- what kind of kernel caching did you have in mind?
I was thinking the same thing! I'm not sure what it would take to make /rom a FUSE based filesystem, to make it bootable. The current boot process involves the bare kernel mounting Squashfs to find it's init=/etc/preinit & booting from there[1].
Would love some theorycrafting on possible ways to work with DwarFS being a FUSE filesystem.