Hacker News new | ask | show | jobs
by isomorphic 3651 days ago
I agree; no checksumming of user data is very disappointing. If there were performance issues, they could build checksumming into the filesystem, but make it a volume-specific option. No checksumming on the watch, strong integrity guarantees on the Mac Pro.

Their filesystem goals are in some ways consistent with Apple's (marketing) vision: Users would never have terabyte libraries of anything, as the various iServices would (should) be hosting that stuff in the cloud (where one presumes it is stored on a filesystem that actually includes data integrity). Since users won't be storing much of anything locally, Apple needn't care too much about data integrity. This is of course, nonsense.

The idea that Apple's storage devices are error-free is arrogant--but even assuming that were true, there can still be bit errors in the SATA/PCI bus, errors in memory, race conditions, gamma rays, etc. Apple uses ECC memory on their Mac Pro, so obviously someone still believes that sort of thing is possible.

2 comments

I don't see why Apple couldn't just recommend that their pro users who have need of this sort of data integrity locally run their own server with FreeBSD + ZFS. Apple has really backed off on their attempts to market OS X Server to this crowd. Heck, they're probably using FreeBSD already if they need that much data integrity.
Here's the thing: everybody needs this sort of data integrity.

Literally nobody wants their files to be silently corrupted. ZFS made it much easier for (nerds like us) to attain very high levels of data integrity.

APFS was (and maybe still is?) a chance to make that the default for regular people.

Do checksums actually need to be in the filesystem, though? It does seem like an important feature, but couldn't they be done at a higher level, like the way Spotlight indexing works on the Mac today?
It isn't just pro users.

With TB file systems, assuming you haven't outsourced everything to iCloud, data integrity matters. If you have, now you're trusting them not to screw up, ever.

From the movie or mp3 that mysteriously no longer plays, through to more important things - business data or family photos. I suspect many people have experienced bit rot, even if they don't recognise it as such. We've even reached a point where with quoted drive figures copying 2tb from one drive to another will likely result in a bit flip (source - Ars ZFS+btrfs article a couple of years back).

Heck, most people have some level of data loss from a HDD or flash drive fail. Sometimes even when they tried to do all the right things. Only question is whether it was backed up. In the case of personal users, unlikely. Self healing could have been quite some selling point!

I have experienced many bitrotted mp3s in my day. Thankfully I've been able to replace them online. As for other files? I can't recall any that are now unable to open for mysterious reasons.

I also happen to run a home file server on FreeBSD + ZFS, though I don't think that machine has ECC memory so it is still technically vulnerable to corruption.

I hear they use RHEL nowadays.
Does it not matter anyway though? If the file lives locally for a while, and it rots there, the corrupt version will be synced back into the cloud and the corruption will spread. I admit the window of corruption will be smaller, but it will still be there, no?