[nu11ptr]

I've always thought it odd that we don't use ECC as a standard on all computers. I think people really downplay the impact of memory issues. They can be devastating, especially over time. They will slowly corrupt your file system, documents, binary files, code, everything. You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. You are then left with this "icky" feeling. What parts of your work are compromised? Obviously this is far worse with things like binary files and structures than those built on readable text. I have normally just rebuilt my system from scratch in these instances (after fixing the issue of course), but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time. You might not find them until years later.

[inigyou]

This fact comes mostly down to Intel choosing to use ECC support for market segmentation. If you care about your data, you buy the server chips that cost double what the desktop chips with ECC support lasered out cost.

[flaunf221]

> You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512.

And 9 times out of 10 (probably much more) it will be faulty storage, not RAM.

I'd say you are way overestimating impact of no-ECC.

[piguin]

You need the ECC RAM for your filesystem with integrity only then can you be confident you aren't going to be familiar with a concept like this happening 9 or 10 times.

[flaunf221]

You can read online how often correctable errors happen to people with ECC and reporting.

Single digit - sometimes none - errors per year in systems with consumer amount of RAM.

[etbe]

Systems with "consumer amounts of RAM" are often run with what you might call "consumer amounts of sysadmin knowledge" and not notice the errors.

One time I took a working workstation from my home to my parents' home to give them. When I booted it up it gave a lot of strange kernel errors which I googled and discovered were reports of ECC correction. I reseated all DIMMs and it was fine. If that gave errors a few times a year instead of thousands of times on the first boot I wouldn't have noticed.

[inigyou]

Sometimes I get a few per day. I only know that because my Linux kernel configuration happens to have whatever settings enabled to get them reported in dmesg.

[creatonez]

...No? Faulty storage will almost always show up as I/O errors, modern storage generally doesn't want to send a block of data to the CPU if there is an error in it. If your bits are flipping, chances are it's RAM and not storage. Your storage already has ECC.

Some crappy flash drives might not do this, though. Seems to behave like a floppy disk at times.

[etbe]

The majority of storage errors I've seen since using BTRFS and ZFS everywhere have shown up only as the storage returning corrupt data and claiming it to be good. I've seen drives and SSDs return dozens of errors per month for months on end without any SAS or SATA errors or any SMART errors.

I've seen this in DC grade SSDs, NAS grade hard drives, and SSDs sold in high end laptops.

[creatonez]

This can happen, after all HDD firmware and the link is not flawless. But have you actually ruled out RAM/CPU as the primary cause? I'm skeptical.

[auspiv]

This is/was classic market segmentation. Want ECC? Pay up for Xeon (server/workstation CPU)

[scheme271]

AMD has historically supported ECC on all it's chips. That's changed in the last gen or two where some of the lower end APUs don't support ECC but the Pro APUs do and all the ryzen stuff (5, 7, 9) support ECC if you get a motherboard that supports it.

[mixmastamyk]

Theoretically, though I’ve not seen any certified builds. At least advertised. Do tell who offers an ECC laptop PC at a reasonable price.

[scheme271]

Laptops are pretty much unavailable with ECC especially with soldered memory becoming common. On the desktop side, there's multiple motherboards that explicitly note that they support ECC memory. AsRock and Asus both include ECC memory on their QVL lists. I'm not sure how many ITX boards support ECC though so getting it on something that's not a tower might be tough.

But it's definitely possible, I have an ECC setup on my desktop and it's caught a memory error or two over the last few years.

[mixmastamyk]

Even with the desktop I want to see “ECC guaranteed” on the box, not a pile of parts that “should” work.

[scheme271]

It's hard to get more ECC guaranteed than the mobo manufacturer having ECC ram on their QVL (qualified vendor list) for the motherboard and indicating that ECC is supported in the specs. I suppose you can just get a complete workstation with ECC from HP or DELL and that'll be guaranteed to work but you'll certainly pay a hefty premium for that.

[sliken]

Sort of, often there's workstation chips, like the xeon E3, which is a warmed over desktop chip ... with ECC. Price difference when I bought was $20 CHEAPER for the CPU (at the cost of 100 MHz or so). I did pay $50 more for a motherboard and 10-15% more for ram.

Or just get an AMD Ryzen.

[zerof1l]

It's very important IMO that the ECC failures get reported and logged. The DDR5 on-die correction is not true ECC; the correction happens silently and is not reported or logged.

I have DDR4 ECC memory on my home server. Last week I got a crash... logged into my BMC and saw that one of the modules had "uncorrectable ECC". Then I looked over the logs and saw about 200 or so errors for the exact same module but "correctable ECC". They would happen over a short period of time followed by 4 months of no issues. Thanks to ECC and logging it was quick and easy to catch the issue right on the first occurrence, no data corruption. I have 4 modules in total, and only this one was having issues, so clearly a hardware fault.

Now imagine it was a regular non-ECC memory. There won’t be any crashes due to uncorrectable ECC; another 3 or more years would pass before corruption would become frequent enough for the issue to be noticed. At that point, there would be hundreds of corrupted files.

[TacticalCoder]

> I think people really downplay the impact of memory issues.

I don't think so. I've got my important data backed up on offline HDDs and SSDs and my most important data lives on a RAID system (ZFS) on a Xeon with ECC RAM but...

... there are simply a huge lot of checks everywhere making it so that bit flips on non ECC systems simply aren't that serious.

We live in a world of checksums and retries at every layer and distributed systems that are just about everywhere.

I went on for decades without ECC memory and nothing bad happened. Yes, I had the occasional bitflips. I did found out faulty mem using memtest (or weirder stuff, like some RAM stick identical to the other three, same chip everywhere, same sticker: but reported by memstick as being some other mem).

But I don't think that the "one bitflip cascading into corruption of your life's data" is something necessary to freak about.

It's not happening. Never happened to anyone.

If you (or other) are really concerned, you can use my scheme:

    - dsc09874.jpg

becomes:

    - dsc09874-b3-73b0835221.jpg

which means "that file's Blake3 checksum begins with 73b0835221" (use something else than Blake3 if so desired).

About 1/3rd of all my files are "tagged" this way, directly in the filename: this survive different medium and different OSes.

I then have my custom rsync backuping script doing a dry-run and warning me in a "stop the world!" fashion if it detects a bogus checksum (either on the source or on the dest: during the dry-run any modification to a file with a checksum is verified and any checksum mismatch means "stop the world").

I then also have, in addition to my custom backuping script, another script that allows me to verify either all files with that naming convention or a percentage I pick of those files (so basically some random sampling).

It's not mutually exclusive with ZFS / a system with ECC RAM.

> but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time

I hear you but... How much do you really need, years later, a file that you haven't needed for years? That one file is unlikely to be a big loss.

Now I'm not saying you're wrong: I ended up with my naming scheme (appending a partial cryptographic hash to the filename) after finding a series of about 20 pictures that were corrupted. And on all the backups I first checked, they were also corrupted. I ended up finding a very old data CD on which that series of .jpg files were fine. And I decided "never again, I want such SNAFU to be detected in the future".

But truth is: with 40 000 pictures, life without those 20 pictures would still be fine.