[in_vestor]

Where do you get the idea that HDDs only retain data for 5 years? The physics I learned in college suggests they will retain magnetic domains for hundreds or even thousands of years.

Yes they can fail mechanically (is this what you mean?) but you don’t necessarily lose your data.

[adrian_b]

They will fail mostly mechanically, but also the very small magnetized bits of modern HDDs will flip spontaneously at normal temperature after a much shorter time than "hundreds or even thousands of years" (because the energy needed to flip a small bit is not large enough in comparison to the thermal fluctuations to make the flipping probability negligible).

Many of these bit flips, but not all, will be corrected when the sectors are read, due to the error-correcting codes that are used in HDDs.

This is not theory, I have stored data for several years on more than 60 HDDs of various capacities from both WD and Seagate, most of them being the more expensive models with extended warranty durations, but even so, only few of the HDDs did not have any non-correctable error after several years. (Fortunately I was careful to use redundancy, so there was no data loss.)

Moreover, some of the biggest HDDs that are available now are no longer suitable for long term data storage, because in order to improve the performance they store metadata in a flash memory, which has a more limited data retention time.

After more than 5 years the complete loss of a HDD should be expected at any time, but even after 2 or 3 years a few non-correctable errors are probable.

When a HDD fails mechanically, one might pay a data recovery service, but that might have a price similar to a new HDD, so if you plan to not replace your HDDs often enough with the hope of using data recovery, it is pretty much certain that the cost will be much higher than replacing any HDD preemptively when its warranty expires.

[in_vestor]

Have you truly observed this type of degradation?

I do archive work and have 20+ discs from the 2010 era. Mostly the first generation of PMR drives. I have never had any data degradation problems.

You can also find lots of YouTube videos of people spinning up drives from the 80s and 90s which still hold their data without problem.

More scientifically, the phenomenon you talk about is modeled by the Arrhenius equation (1), where the activation energy to flip a grain is given by KuV/KbT, where Ku is the anisotropy of the magnetic media, V is the volume of a grain, Kb is the Boltzmann constant, and T is temp in Kelvin.

HDD manufacturers engineer this ratio to be >60 (usually targeting 70-90 to be safe). Media manufacturing is imperfect, so there is a log normal distribution of grains on real-world media, but if we assume that 60 is the energy barrier for all grains, a KuV/KbT of 60 would mean it takes 362 million years for half the grains to flip, assuming an attempt frequency of 10^10.

Where is my math wrong?

(1) https://en.m.wikipedia.org/wiki/Arrhenius_equation

[adrian_b]

Your math is probably right, but a modern HDD has more than 2^19 data bits.

Assuming that your computed time is right, that means that there is a 50% probability that one bit of a HDD will flip after less than a week.

Most such bit errors will be corrected when a sector is read and the controller will rewrite a bad sector with a valid value, so the bit errors will not be cumulative in normal usage.

However when the data is stored for years without powering up the HDD, the bit flips will accumulate and they may pass the threshold needed to cause an non-correctable error.

While I do not remember to have ever seen non-correctable errors on the HDDs that I have been using daily, on identical HDDs that have been stored for years without being powered up I have frequently seen both cases when the drive reported non-correctable errors and cases when the drive reported no error but the file hashes used for error detection identified corrupted files.

The older HDDs with low data capacities had much longer lifetimes, but also the perception of those claiming that data has been stored OK on them may be wrong if they have not used any means to detect the corrupted files, because even if the HDD reports no errors, that is not good enough.

[in_vestor]

One point of clarification: one bit on a classic PMR drive contains hundreds of magnetic grains. It is the grains that flip, not the bit. It would take many grain flips to affect the bit. Errors of this sort do not manifest as flipped bits per se—they manifest as a degraded signal, which the drive may or may not be able to translate to the correct bit sequence correctly. Also, the nature of ECC is (usually) that you get the correct sequence or an error. It would be unusual to get an incorrect sequence unless that is happening somewhere off-drive.

If you have a stored drive that is reporting errors, my starting assumption would be that something else is causing problems besides the platter—maybe the heads have gotten a bit of corrosion from humidity.

Still disagree?

[adrian_b]

Because the HDD manufacturers avoid to provide the information that would be necessary to estimate with any degree of certainty the data retention time for HDDs, we cannot know for sure the causes of HDD errors during long term storage, so we can only speculate about them.

Nevertheless, the experimental facts, both from my experience during many years with many HDDs and from the reports that I have read are:

1. Immediately after the warranty of a HDD expires, the probability of mechanical failure increases a lot. I have seen several cases of HDD failures a few months after the warranty expiration, while I have never seen a failure before that (on drives that had passed the initial acceptance tests after purchase; some drives have failed the initial tests and have been replaced by the vendor).

Therefore one should never plan to store data on HDDs beyond their warranty expiration.

2. When data is stored on HDDs that are powered down for several years, one should expect a few errors (I have seen e.g. about one error per 2 to 8 TB of data), which cause either non-correctable errors or wrong corrections that corrupt the data.

The effect of such errors can be easily mitigated by storing 2 copies of each data file on 2 different HDDs.

An alternative is to introduce a controlled data redundancy, e.g. of 5% or 10%, with a program like "par2create".

That works fine against wrongly corrected sectors, but when a non-correctable error is reported, many file copy programs fail to copy any good sector following a bad sector, so one may need to write a custom script that will seek through the corrupt file and copy the good sectors, in order to get enough data from which the original file can be reconstructed.

Storing everything on 2 HDDs, preferably of different models, is the safest method, as it also guards against the case when one HDD is completely lost due to a mechanical defect.