Hacker News new | ask | show | jobs
by soulbadguy 3629 days ago
Do you need to transmit 500GB every time or just a diff from a previous dataset ? If it's the later case, using the send/receive functionality of a file system with snapshot and incremental backup (ZFS,BTRFS etc..) can be significantly faster than using pure rsync. Rsync would needs to scan the complete 500GB of data to find the blocks to send, while send/receive can compute the diff much faster
1 comments

Negligible W/R/T transfer time. While using native capabilities of ZFS is awesome, you're now locked into a particular FS at both sides of the transfer (this may or may not be an issue).

(Also, there's a patch for rsync that allows you to force computing the checksum in batch, not on each invocation; but that's getting into hairy territory that's rarely needed - I used it in exactly one case so far)