[cyanmagenta]

The problem I have with ECC discussions is that it’s always anecdotal. There is one Google study finding memory flips with some probability in their datacenters, but it was awhile ago. Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips. That’s not a knock on this article per se, but it’d be nice to have actual data here.

[adrian_b]

As TFA says, it is very likely that the hyperscalers continue to make studies about DRAM reliability in their servers, but they do not publish them, either because they believe that such data could be useful for competitors, or, more likely, they might have NDAs with the memory vendors.

When you do not own a great number of servers, you cannot provide anything else except anecdotal evidence.

Because the vast majority of PCs do not have ECC memory, even the companies that have a great number of PCs have no idea about the reliability of the memory used in those PCs, because it is very difficult to distinguish memory defects from the huge number of software bugs.

Moreover, the frequency of memory errors is proportional with the amount of memory you have, so those with tiny amounts of memory, like 8 GB, are much less likely to encounter memory errors than those who have 32 GB or 64 GB of DRAM in their PCs.

I have tried to always use ECC memory in my computers, whenever possible. Even my laptop is an older Dell Precision, with ECC memory.

But I can also offer only anecdotal testimony.

I have also seen cases like that mentioned in TFA, where the existence of ECC allowed to discover that some modules were not seated well in their sockets, so reseating them prevented the reappearance of errors.

I have also seen a case when a laptop was not used for a long time and it was stored in a rather humid place, so the contacts in the SODIMM sockets had oxidized, which resulted in frequent memory errors. Scrubbing vigorously the contacts and reinserting the memory modules solved the problem.

I have also seen many cases where certain memory modules degraded after many years of use, e.g. 5 or more years of 24/7 use, and they began to have frequent errors, e.g. multiple errors per day, even if when they were new the error rate could have been of one error per year or even less.

In such cases ECC was extremely useful, because the DIMM that had become worn out could be identified and replaced and the server could work fine some more years.

[phkahler]

>> Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips.

That's because the on-die ECC cannot detect errors in the transmission of data from the DIMMs to the CPU. It seems like on-die ECC is meant to "hide" some level of errors to make less reliable RAM chips appear good. I have no idea if that's a reasonable way to do it, but it seems like it could be misused. Like TFA suggests, we need to know how the two levels of ECC interact.

[cyanmagenta]

Right. Everyone agrees that real ECC is more robust, and production servers of course run it. But it seems to me that if most RAM bitflips are in the module itself (rather than in transmission to CPU), then on-die ECC could in theory make the probability of a bitflip in practice so low that for things like home servers it isn’t worth the price premium for real ECC. But without data on this, everyone is just speculating.

[adrian_b]

No.

On-die ECC was added in DDR5 because the error frequency has increased too much.

Its only effect was that now DDR5, as seen externally, has a reliability similar to that of the older memory generations. On-die ECChas not brought any serious improvement in reliability. It has just prevented the degradation of the reliability.

The errors generated internally are caused mostly by the ionizing radiation from the environment, which discharges the storage capacitors.

However there are many errors that appear during the communication between memories and the CPU, which are caused by the electrical noise from the environment.

These external errors are much less likely to appear for soldered DRAM. Because of this, for soldered LPDDRx memory it is less important to have end-to-end ECC than for socketed modules, which are much more vulnerable to noise.

The DDR5 standard has some options for using some error detection for the communication link, but it is impossible to know whether a given computer or motherboard has implemented such options or if they are enabled by the firmware when the motherboard has the physical support.

The only way to be certain that the memory works fine is to use ECC that covers completely the circuit from the CPU memory controller to the values stored in the DRAM and back to the CPU memory controller and your operating system has the appropriate device driver for obtaining the error reports.

[aidenn0]

Besides electrical noise, there are mechanical issues as well. I've seen systems fail memtest86+ after being moved, and the issue being fixed after reseating the DIMMs. ECC can catch mis-seated DIMMs very quickly.

[adrian_b]

The mechanical issues also act through increasing the sensitivity to electrical noise.

The imperfect contacts are equivalent to adding series resistors, possibly in parallel with parasitic capacitors, on the link traces, allowing a greater amplitude for the noise pulses.

[hnuser123456]

In addition to reducing the energy content of the actual desired signal.

[Marsymars]

Also to note, asides from the physical differences, LPDDR5+ also features link ECC.

[zargon]

This is the attitude that needs to change. If RAM is 25% of the system cost (say for a home server in your example), then the cost for ECC is 3% of the system cost. There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff. ECC should be standard, period.

[Wowfunhappy]

When I last looked at this (a couple of years ago), it wasn't just the cost difference—the ECC memory available to purchase was also slower than non-ECC offerings, or at least cost significantly more if I actually wanted the same speed. At least for me, going ECC simply didn't make sense.

Caveat that I was building a gaming PC at that time, I might have gone differently for a home server. And, this was before the memory market went nuts for unrelated reasons.

[adrian_b]

Yes, ECC DIMMs are available only for the memory speeds guaranteed to work by the manufacturers of CPUs, like Intel and AMD, e.g. for up to DDR5-5600 in UDIMMs or DDR5-6400 in RDIMMs.

The "gaming" DIMMs are available in much greater speeds, which are "overclocked" speeds, i.e. there exists absolutely no guarantee from Intel or AMD that they work at the advertised speed, even if they frequently do work.

However, with such overclocked DIMMs, you do not know how well they work. They might work most of the time, but e.g. in a hotter room they might have an error or two per day without you noticing, especially if the computer is used mainly for games.

For someone doing anything professional on a computer, overclocked DIMMs are not an acceptable choice.

[jmb99]

> ECC DIMMs are available only for the memory speeds guaranteed to work by the manufacturers of CPUs, like Intel and AMD, e.g. for up to DDR5-5600 in UDIMMs or DDR5-6400 in RDIMMs.

This is not true[1]. These DIMMs are not guaranteed to work with all Threadripper CPUs, but they are guaranteed themselves to be stable at their advertised frequencies, which are higher than JEDEC. (And from my experience, 9000-series threadripper is perfectly happy to drive 8-channel 1Rx8 at 7200MT/s.)

AMD (and possibly Intel, I don’t have any experience with their recent offerings) also lets you overclock memory completely on WRX90. So if you buy JEDEC sticks, you’re welcome to try to push them as hard as you can.

[1] https://v-color.net/products/ddr5-ocrdimm-amd-wrx90-workstat...

[zargon]

AMD, Intel, and motherboard manufacturers do qualify the CPUs and motherboards for "overclocked" (above JEDEC) memory speeds. The systems are designed to work at the overclocked transmission speeds. They need to be paired with memory modules that are capable of functioning at those speeds. With ECC presently being a datacenter thing, companies only make it according to JEDEC specs. If ECC were standard for everyone, then the pre-qualified overclocked "gaming" memory modules would be available in ECC configuration as well.

Without ECC, nobody knows if their memory is working or not. Regardless whether it is running a standard JEDEC speed or an overclocked one.

[undersuit]

So as I was overclocking my AMD Ryzen 7 2700 I discovered that the Infinity Fabric has ECC because if I tried to run my fancy DDR-3466 ram at full speed I started getting logged errors from it. Those failed transactions slow down the system because the data needs to be corrected or even re-sent. The RAM runs fine overclocked at DDR4-4200 in a newer system, but my old Ryzen 7 2700 couldn't handle anything past DDR4-3266 without the errors creeping in.

Without the reported ECC on Infinity Fabric I would have had no idea that the errors were happening and affecting performance.

[jmb99]

There are absolutely fast DDR5 ECC DIMMs[1]. Just break out your wallet (and piggy bank, and rainy day fund, and..)

The main reason there aren’t more options here is that a) high-frequency memory usually isn’t sufficiently stable, unless binned, and b) there’s very, very, very little demand.

[1] https://v-color.net/products/ddr5-ocrdimm-amd-wrx90-workstat...

[Wowfunhappy]

Right, my point was that it's no longer 3% of the system cost.

[pseudalopex]

Intel and AMD segmented the market. Workstations and servers use registered memory. Desktops use unregistered memory. Most desktop motherboards or CPUs do not support ECC. And most people who want ECC want reliable more than fast. So ECC unregistered above standard speed memory is made and priced for a niche. But this would change if ECC was standard.

[justsomehnguy]

The same reason your utility trucks isn't rated for 200KM/H speeds.

[toast0]

Let me kmow where I can get ECC RAM that's only 12% more than the price of non-ECC and I'll strongly consider it the next time I do a build. (Otoh, prices are high, so I'm not building anything for a while unless stuff breaks)

[etbe]

For second hand gear RDIMMs are significantly cheaper than UDIMMs. As all RDIMMs have ECC that means generally ECC is cheaper for second hand systems.

Aliexpress had $50 single socket motherboards for Xeon E5-26xx CPUs and $150 dual socket motherboards last time I checked that use DDR4 RDIMMs.

Before the prices went up due to AI foolishness DDR4 RDIMMs were $1.20AU/G means means something like $US0.80/G.

[adrian_b]

Some years ago, it was easy to find ECC modules with a so small price difference.

From that time, I have a few old computers with 64 GB or 128 GB of DDR4 ECC memory.

Unfortunately, after the passage to DDR5, DDR5 ECC UDIMMs have become hard to find, and even when you could find them, the price difference could be as high as 30% to 50%.

[tbrownaw]

> There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff.

It is if you're doing something where corruption is detectable after the fact, and happens rarely enough that redoing affected work adds less than 3% overhead.

[zargon]

If you don't have ECC, you don't even know that your CPU is executing the correct code. There goes the "detectable" part.

[tbrownaw]

There's a decent amount of distributed stuff that has a design principle of not trusting the hardware it's running on.

[fvwqcecvq]

DRAM is just one level in the memory hierarchy and it can absolutely be tested by the other levels in the system. It won't be continually monitored, and there will be a performance penalty, and it's a PITA, but it can definitely be done without hardware ECC. Even some of that can be minimized as most tests are statistical in nature anyway.

[doctorpangloss]

i have years and terabytes of ECC DDR4 and DDR5 memories, UDIMM and RDIMM, with edac enabled all this time, and i have literally never once observed a single error correction. however, (1) i built all these computers myself (2) the servers are located on the middle floor a thick, concrete multi-story warehouse-to-office conversion. as others have said, on a correctly built computer, the source of bit flips is cosmic radiation, and there is a lot of mass between these machines and the sky.

[SnowflakeOnIce]

It's not just cosmic radiation or improperly built computers that result in bit flips. Row hammer is a thing :)

[doctorpangloss]

anyway i was completely wrong and a few nodes had EDAC events haha