Hacker News new | ask | show | jobs
by zeotroph 685 days ago
That only seems to work on btrfs, XFS, (and maybe now or very soon) ZFS and bcachefs: "[duperemove] simply finds candidates for dedupe and submits them to the Linux kernel FIDEDUPERANGE ioctl." [1] (aka BTRFS_IOC_FILE_EXTENT_SAME), and this ioctl "performs the 'compare and share if identical'" (and locking etc.) work [2]. But on those filesystems, that is a nice feature, plus it lets the tool get away with a weak hash like murmur3.

1: http://markfasheh.github.io/duperemove/duperemove.html 2: https://manpages.debian.org/bookworm/manpages-dev/ioctl_fide...

2 comments

A fallback to hardlinking would work for a lot more filesystems, though of course it would be limited to files that are completely identical rather than any identical extents / blocks within files.
More importantly, hardlinking means writes to one file would affect both, while FIDUPERANGE creates copy-on-write semantics.
Quick note : I removed murmur3 support, and replaced it with xxhash