Can tmpfs be backed by persistent storage? Most of the recent stuff I've worked on is a little too big to fit in memory handily. Ideally about 20GiB of scratch space for 4-8GiB of working memory would be ideal.
I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster
tmpfs is backed by swap space, in the sense that it will overflow to use swap capacity but will not become persistent (since the lack of persistence is a feature).
We built EtchaOS for this use case--small, immutable, in memory variants of Fedora, Debian, Ubuntu, etc bundled with Docker. It makes a great CI runner for GitHub Actions, and plays nicely with caching:
Why do it at the block level (instead of tmpfs)? Or do you mean that you're doing actual real persistent disks that just have a lot of cache sitting in front of them?
The block level has two advantages: (1) you can accelerate access to everything on the whole disk (like even OS packages) and (2) everything appears as one device to the OS, meaning that build tools that want to do things like hardlink files in global caches still work without any issue.
I've had good success with machines that have NVMe storage (especially on cloud providers) but you still are paying the cost of fsync there even if it's a lot faster