Hacker News new | ask | show | jobs
by esseph 2 days ago
> Just last week, I spent several hours reading the rsync C Language source code line-by-line so I could write a custom patch to add nanoseconds to the file creation dates.

Pause

I need to hear about this :)

What is your use case, if you don't mind sharing?

2 comments

>What is your use case, if you don't mind sharing?

I have custom utilities that detects moved/renamed files by analyzing heuristics of filesize, modification times, and creation times. It can work with files' timestamps without any fractional seconds (nanoseconds) but the most reliable and confident heuristics takes advantage of nanoseconds to build the most accurate hashtable of files.

The latest rsync 3.4.4 will copy creation times but it didn't copy over the nanoseconds. It does copy the nanoseconds for file modification times but not the creation/birth times.

Command line file sync tools like rsync or robocopy can't detect file moves which means they will blindly delete files from the old spot and re-copy them to the new spot. Imagine a directory that's 2 terabytes full of files that were just renamed. Those utilities will delete 2 terabytes then recopy another 2 terabytes. It's a risk of losing 2 tb of files if there's a interruption and needless SSD wear & tear.

I may send my changes up to the rsync github repo. I just don't have an active account at the moment.

OK, you may be right.

I have always verified that rsync copies the modification timestamps with nanosecond resolution, because I use these timestamps, but I have never looked at the creation times, because I do not use them.

I always have content hashes attached to files in an extended attribute.

For detecting duplicates with different names I always use the content hash, which is much more reliable than a creation timestamp and it is also useful for other purposes, e.g. for detecting file corruption.

I cannot see the original comment as it was deleted, but it seems very weird.

There are indeed a lot of Linux/UNIX programs that lose metadata when copying or archiving files, like losing extended file attributes or the high-resolution timestamps with nanosecond resolution.

The filesystems that I use (XFS on Linux and UFS on FreeBSD) have timestamps with nanosecond resolution and also extended file attributes, so I was also annoyed in the past when discovering the many utilities that lose metadata.

However, rsync is one of the few utilities that preserve all metadata even when copying between distinct file systems, e.g. between UFS and XFS.

That is why for many years I have never used anything else except rsync over ssh, to be sure that the metadata is intact.

However, to preserve metadata no Linux utility may be used with default options.

For instance, I alias rsync to '/usr/bin/rsync --archive --xattrs --acls --hard-links --progress --rsh="ssh -p XXX -l YYYY"'

Therefore I wonder on which filesystem the previous poster has seen that rsync loses metadata, because this does not match my experience.

In the past, I have also patched some open-source programs to preserve the high-resolution timestamps, but rsync was OK since the beginning.