[huhtenberg]

How did you know it was a change at rest?

The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the disk to another machine and re-read again. If it doesn't match, only then it's an actual at-rest bitrot... OR it's a drive's firmware bug, because corrupted data must be corrected or it must not be returned at all.

[notacoward]

> How did you know it was a change at rest?

Because we had the checks for it in flight. Also, more often than not these same blocks had been checked before, and found to be fine.

> The only correct way to test for bitrot is to read the data back immediately

No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot.

> must be corrected or it must not be returned

Every error-correction technique has a limit to how many simultaneous errors it can correct. Beyond that, bits can be flipped in a way that seems valid but in fact is not (detectable by cross-checking with other erasure-coded fragments of the same block on other machines). Just because you haven't seen it doesn't mean it doesn't happen. As I said, and as others have said many times, with sufficient scale and time even the most unlikely scenarios become almost inevitable. Why do you persist in telling me I didn't see what I saw with my own eyes? Are you assuming that my thirty years in storage gave me less understanding or insight regarding these issues than whatever experience (if any) you have?

[huhtenberg]

>> The only correct way to test for bitrot is to read the data back immediately

> No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot.

Well, no. If you want to check for at-rest bitrot, you need to make sure that you've written out the correct thing. Otherwise it's not possible to tell at-rest corruption from the one that happened on the way in.

> Every error-correction technique has a limit to how many simultaneous errors it can correct.

But it can detect that the case when it can't recover. Which is why it will either produce a correct output or an error.

> As I said, and as others have said many times, with sufficient scale and time even the most unlikely scenarios become almost inevitable.

This is not an argument if it goes against how things actually work.

> Why do you persist in telling me I didn't see what I saw with my own eyes?

I am merely curious in your exact testing technique, because at-rest bitrot is vanishingly impossible, even at the exabyte scale. For it to happen, the data and its ECC (7-11% of the data size) need to be both corrupted in a coordinated way. That is exceedingly unlikely. Especially in the context of academic papers that found that on-disk corruption is nearly always clustered and is either small scale or full-sector failures.

So when you say you ran into a lot of these cases, it's only natural to ask for details. And "scale" is not a detail.

> Are you assuming that my thirty years in storage gave me less understanding or insight regarding these issues than whatever experience (if any) you have?

I have no way to tell. But given your experience, can you explain how at-rest bitrot, should it occur, can seep through the on-disk error correction? I am not talking about raid-style setups, just the banal ECC record in a disk sector [1].

[1] https://en.wikipedia.org/wiki/Advanced_Format#Overview (linking to Advanced Format, because it has a diagram)

[notacoward]

> But it can detect that the case when it can't recover.

That is simply not true. For any parity/ECC/FEC/erasure-code scheme carrying M data bits in N (greater than M but less than 2M) total, there must be multiple data patterns that will match the same error checks. That's just mathematics. Also, bear in mind that ECC bits can be corrupted too. This opens up the distinct possibility of something that looks like a correctable error, but the "correction" leads to a wrong result. I've seen such issues in many kinds of storage systems, from low level to high. Anyone who has actually worked in this area, instead of deriving their "expertise" from a quick scan of Wikipedia, would be utterly unsurprised by the idea that disk firmware might do such a thing, or have bugs in their ECC implementation, or not follow a spec.

Whatever the causes, whatever the merely-theoretical probabilities, the fact remains that I've seen these. I've been paged for them. I've done the analyses of possible causes. A bit pattern was written and repeatedly verified over a quite long period of time (ruling out data path issues), then at some point a different bit pattern was read and would persistently be read thereafter. How is that not real bitrot? How does it matter, beyond ruling out everything above the disk level, what the precise causes are? If you can't answer those questions, you're just posting noise.

[huhtenberg]

> That is simply not true.

It indeed is not. Had to reread the theory and I stand corrected, RS-style ECC can't detect errors in excess of the redundancy count.

> How is that not real bitrot?

It is and I can see how it can happen.

> How does it matter, beyond ruling out everything above the disk level, what the precise causes are?

It would've mattered if a drive could detect on-disk bitrot reliably, which was what the stats I worked with (also in exabytes, funnily enough) and the IEEE papers I read led me to believe.

For what it's worth, you won. Hats off.

[notacoward]

Thank you for an interesting (despite being contentious) conversation.

[jeffbee]

Didn't you answer your own question above? It's firmware bugs. The disk reported a successful write at block X but it actually wrote block Y. Later you read block Y and you get data X. The block-level ECC codes are consistent. You also stand a low but not zero probability that you requested a read at block X and were served up some other block, again with matching checksums. And of course there's always the possibility that your firmware simply has a bug in the code checker.

The paper "Parity Lost and Parity Regained" assigns a probability of 1.88e−5 to misdirected writes bugs among disks, so if you have a warehouse full of disks you now have this nightmare.

[notacoward]

Fun question: what if a relocation table gets corrupted? And what protection is there against that possibility? You can bet it's not the same ECC as on data blocks. The rest is left as an exercise for the reader. ;)

[jonah-archive]

uh oh, I recognize this one. Love to have a file corrupt after months at rest with no access logged and no mtime changes because a file on a neighboring track needed rewriting (SMR, of course).

[booi]

ZFS and other checksumming file systems can detect bit rot in data at rest. When data is read back, that sector is checksummed and compared against when it was written before returning the request.

You can periodically scrub the entire pool to find and even fix these issues (in a pool with redundancy)

[legulere]

The bit rot could have still happened during writing though, if the bitrot is found in the first scan.

[huhtenberg]

Sure. However the main purpose of scrubbing is to flush out deteriorating media and to prompt the drive to relocate salvageable sectors and to report completely dead ones.