Hacker News new | ask | show | jobs
by hachari 2028 days ago
Why not use BTRFS with file deduplication and transparent compression (zstd specifically)?
2 comments

This is a read-only file system, so it’s able to exploit certain properties of that—- locating similar files next to each other, for example.
I don’t see how read only helps at all.

Btrfs can dedupe at the block level.

Consider how much of work in btrfs is done to just handle the case of modifying existing files—or reducing file system size.. It is basically the reason it uses b-trees. It's in the name!

For example, when dedupping in block level it needs to know (right?) how many times a block is being used, so it can be collected when it runs out of uses.

ISO9660 can also express dedupped (hardlinked) files with the Rock Ridge extensions. I don't know but I'm wondering it could even do block-level dedupping if the generating program abused the format a bit..

A compression benchmark of both filesystems would be of interest in this regard (lzo, zstd and zlib), both read speed and compression wise
Sure, but it sounds like block deduping is only one of several optimizations that DwarFS is able to take advantage of because it’s a read-only FS.
Is Btrfs stable yet?
For most use-cases, yes. But not if you plan on using raid5+ [0]

[0]: https://btrfs.wiki.kernel.org/index.php/Gotchas#Parity_RAID

My own experience indicates it's brittle to power failures and will corrupt in annoying ways in the event of a power failure (or hard reboot) as of ~1 year ago.

Year isnt as useful as kernel version.
I was running whatever Arch's Linux kernel was at the time. 5.4 I believe? Arch is pretty much bleeding edge, so year is relevant.
Yup, in Arch even month, week and day is relevant.
Btrfs is default in Fedora 33.
Fedora also has Wayland as default. If Debian had Btrfs as default, that would be an argument.
It's also been the default for the root filesystem in openSUSE since 2012. It also integrates with the package manager - zypper creates pre- and post- snapshots when installing updates, and the snapper utility makes it easy to roll back if your system is busted after installing an update. It's saved my bacon on multiple occasions.

(btrfs also used to be the default for /home, though that changed at some point. When I made a new install last year the installer suggested xfs by default.)

I'm only ever switching to a new filesystem if its correctness has been formally verified.
For which filesystems is that case?
None. But the tools are there, so it should be possible.