[ocd]

I have external parity for everything with rsync. I don't have any storage that isn't matched with external backups. Fortunately I've never seen any bitrot or had a drive fail on me since I tend to replace them before they do. And yeah, I could add in some extra HDDs for RAID, but this is more a personal choice because of how much I hate SATA.

Filesystems like ext4 are because I value boring and rock solid stability over semi-experimental status of modern filesystems with all the features.

[ngetchell]

How do you know you haven't seen bitrot? What rsync arguments do you use to error out when an old, untouched file suddenly changes?

[ocd]

I suppose I couldn't guarantee I haven't, but I keep a total backup, and other backups based on file lists. I keep a record of every file of importance that I've ever written/modified/interacted with, and those lists themselves are also git managed.

I compulsively look at the rsync backup running to see what's going to transfer, and typically do a dry-run first. That's no perfect solution, but it works for me. If I see something odd I don't quite remember, I check the hashes between both drives, and every time it turns out it was just a path change since I try to keep my data as organized as possible.

rsync arguments are just the plain `--archive --acls --xattrs --verbose` and depending on which backup I'm doing, `--recursive`, `--delete`, `--files-from=`. Nothing other than vanilla at all.

[CTDOCodebases]

What do you use for external parity? Par2?

If you were to replace your current drive with a new larger one and your RAM or your SATA controller silently corrupted the files how would things play out? Would these corrupted files propegate to the backups?

Ext4 is battle tested so I understand your reasoning. I think you just need to figure out a way to detect silent corruption and a way to snapshot your files in case they do get corrupted.

From experience I can tell you that it is an absolute pain having to manually sort through a bunch of files trying to detect which ones are corrupt and which ones are good.

[ocd]

Misspoke. I didn't mean that type of parity, but just hashes, and a matching drive (in make and capacity) externally for an exact copy.

[dannyw]

+100 to this. I don’t want to be overtly paranoid but you genuinely are unlikely to notice real data corruption unless you actually checksum.

It is a matter of time when you’re storing a lot of data — not if.

[Brian_K_White]

rsync does not protect you from bitrot unless you have some script that specifically collects the rsync log and tracks when existing files change, and you somehow know every file that should not have changed and get alerted that a file changed when it shouldn't have.

Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.

Ok the tie-breaker data could come from history. Yesterdays log and the day before gives you the extra data points to say which side changed. As long as the log itself is somehow above reproach.

Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change." and you would have to actually do that tagging and reviewing.

[TacticalCoder]

> rsync does not protect you from bitrot unless you have some script that specifically collects the rsync log and tracks when existing files change, and you somehow know every file that should not have changed and get alerted that a file changed when it shouldn't have.

You're right and that's the big thing most are missing. Name your files and write your scripts accordingly I'd say.

My rsync backup'ing script first does a dry run and then serious analyses on the result of the dry run is done. If anything looks out of the ordinary, the script refuses to go ahead and warns me.

rsync silently copying bitrot to backups is the stuff nightmares are made off.

> Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.

I personally have a sizeable percentage of my files (like, say, family pictures) that have part of the checksum in the name of file:

    dsc003784-b3-79a7208c51.jpg

This means that file should have it's Blake3 checksum beginning with 79a7208c51. And these have been vetted, so in case there's a mismatch, the authoritative file is the one with the proper checksum.

And the system/disk that gave back a bogus checksum has explanations to give ; )

> Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change."

I don't think it's a myth. My script does that with the checksum as part of many filenames: if during the dry run the script notices a file with such a checksum is present but different on both systems, I know immediately that something is off.

[bobmcnamara]

What do you dislike about SATA?

[ocd]

The cables and how they have to be managed/positioned in a case. I've had panics over a drive failing when it was just the cable after I'd already replaced the drive, restored data, and ran tests on the old drive that turned out to be fine each time.

[CTDOCodebases]

I have multiple home servers running ZFS and the first thing to check when encountering errors is the SATA cables. Those blue foil fan-out types are the worst from my experience.