I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory."<p>Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the software being stored in that memory being robust against cosmic rays. And when you have thousands of machines with petabytes of RAM, those bitflips do happen. Google wasted many man-years tracking down corrupted GFS files and index shards before they finally bit the bullet and just paid for ECC.
I beg this, every time this conversation comes up it’s the same answer “I don’t see a problem”.<p>It’s so easy to chalk these kind of errors to other issues, a little corruption here, a running program goes bezerk there- could be a buggy program or a little accidental memory overwrite. Reboot will fix it.<p>But I ran many thousands of physical machines, petabytes of RAM, I tracked memory flip errors and they were _common_; common even in: less dense memory, in thick metal enclosures surrounded by mesh. Where density and shielding impacts bitflips a lot.<p>My own experience tracking bitflips across my fleet led me to buy a Xeon laptop with ECC memory (precision 5520) and it has (anecdotally) been significantly more reliable than my desktop.
As someone who has had to read thousands of random game crash reports from all over the interwebs (you know when Windows says you might want to send that crash log? like that), I totally agree.<p>Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when you can't trust your memory you can't trust anything.<p>And as the timeline goes to infinity, you may also get one of these reports and be asked to fix it... good luck.
Good news is that for DDR5, ECC is a required part of the spec and should be a feature of every module:<p><a href="https://www.anandtech.com/show/15912/ddr5-specification-released-setting-the-stage-for-ddr56400-and-beyond" rel="nofollow">https://www.anandtech.com/show/15912/ddr5-specification-rele...</a>
I used to be pretty skeptical of ECC for consumer-grade hardware, mainly because I felt that I'd always prefer cheaper/more RAM over ECC RAM even if it meant that I'd get a couple of crash every year due to rogue bitflips. For servers it's a different story, but for a desktop I'm fine dealing with some instability for better performance.<p>But these days with the RAM density being so high and bitflipping attacks being more than a theoretical threat it seems like there's really no good reason not to switch to ECC everywhere.
About 1/3 of Google's machines and 8% of Google's DIMMs in their fleet suffer at least one correctible memory error per year: <a href="http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf" rel="nofollow">http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf</a>
I would also add that Row Hammer Attacks are much harder on ECC.<p>When I first tried to replicate the row hammer attack I was not getting any results. Turns out I was doing this on ECC. On non ECC memory the same test easily replicated the row hammer attack.<p><a href="https://en.wikipedia.org/wiki/Row_hammer" rel="nofollow">https://en.wikipedia.org/wiki/Row_hammer</a>
“ECC availability matters a lot - exactly because Intel has been instrumental in killing the whole ECC industry with it's horribly bad market segmentation.”<p>Its.<p>There, I finally corrected Linus Torvalds in something. :))
This is one justified Linus rant! My personal history includes data loss twice because of defective RAM, and many more RAMs discarded after the now obligatory overnight run of MemTest86+ (these were all secondhand RAMs - I would never buy a new one without a refund guarantee). My very first "PC" still had the ECC capability and I used it. My own now very dated rant on the subject: <a href="http://wandel.ca/homepage/memory_rant.html" rel="nofollow">http://wandel.ca/homepage/memory_rant.html</a>
D. J. Bernstein (of qmail/daemontools fame) spoke of it over a decade ago as well. <a href="https://cr.yp.to/hardware/ecc.html" rel="nofollow">https://cr.yp.to/hardware/ecc.html</a>
This reminds me of last year we ordered a new $14K server, it arrived and we ran it through our burn-in process which included running memtest86 on it, and it would, after around 7 hours, generate errors.<p>Support was only interested if their built-in memory tester, which even on it's most thorough, would only run for ~3 hours, would show errors, which it wouldn't. IIRC, the BMC was logging "correctable memory errors", but I may be misremembering that.<p>"We've run this test on every server we've gotten from you, including several others that were exactly the same config as this, this is the only one that's ever thrown errors". Usually support is really great, but they really didn't care in this case.<p>We finally contacted sales. "Uh, how long do we have to return this server for a refund?" All of a sudden support was willing to ship us out a replacement memory module (memtest86 identified which slot was having the problem), which resolved the problem.<p>They were all too willing to have us go to production relying on ECC to handle the memory error.
You don't need to look at kernel crashes to speculate about bus and memory errors -- just check the logs on a few systems that do have ecc. Pretty soon you'll see correctable errors being reported.
ECC works if done right. Accessing a memory location can fix bit-flips (ECC is a 'correcting' code). But systems that don't regularly visit every memory location, can accumulate risk. Those dark corners of RAM can eventually get double-bit errors and be uncorrectable. So an OS might 'wash' RAM during idle moments, reading every location in a round-robin manner to get ECC to kick in and auto-correct. Doesn't matter how fast (1M every hour or whatever) as long as somehow ECC has a chance to work.
For those, like me, wondering what ECC is, here's an explanation:<p><a href="https://www.tomshardware.com/reviews/ecc-memory-ram-glossary-definition,6013.html" rel="nofollow">https://www.tomshardware.com/reviews/ecc-memory-ram-glossary...</a>
Is there such a thing as 'software' ECC where a segment in memory also has a checksum stored in memory and the CPU just verifies it when the memory segment is accessed?<p>It would be a lot slower than real ECC but it could just be used for operations that would be especially vulnerable to bit flips. It would also not know for certain if the memory segment of data or the memory segment holding the checksum was corrupted besides their relative sizes (checksum is much smaller so more unlikely to have had a bit flip in it's memory region).
I bought ECC RAM for my laptop and it definitely was about 4x the price. It's valuable to me for a few reasons -- peace of mind being a big one.<p>Bit flips happen and are real. I really wish ECC was plentiful and not brutally expensive!
I don't know if ECC is that important, but reliability of RAM (or any storage) feels pretty crazy to me.
128GB being refreshed every second for a month error requires that the per-bit refresh process has a reliability of 99.9999999999999999% to be flawless.
Considering we are dealing with quantum effects (which are inherently probabilistic), I wouldn't trust myself to design anything like that.<p>Now back to ECC, I'll probably be corrected, but I don't think ECC helps gain more than two order of magnitudes, so we still need incredibly reliable RAM. If we move to ECC RAM by default everywhere, aren't we simply going to get less reliable RAM at the end?
I simply care that my computer executes code perfectly. Let's settle on "one instance of unintended behaviour per hundred years" for that metric.<p>If it needs ECC memory to do that, then fit it with ECC memory. If there are other ways to achieve that (for example deeper dram cells to be more robust to cosmic rays) that's fine too.<p>Just meet the reliability spec - I don't care how.
There was a great defcon talk a while back regarding using ECC. The concept was called "dns jitter"<p>Basically you can register domains using small bit differences for domains and start getting email and such for that domain<p>If I recall correctly the example given was a variation of microsoft.com<p>All because so much equipment doesn't use ECC
Well shit.<p>I run some large ML models in my home PC and I get NaN's and some out of range floats every month or so. I have spent hours debugging but doing the same computation with the same random seeds does not recreate the problem.<p>How about GPU's and their GDDR SDRAM? Do they have parity bits?
Seems likely that “bad ram” was the reason for the recent AT&T fiber issues, given that 1 bit was being flipped reliably in data packets [1]<p>[1]: <a href="https://twitter.com/catfish_man/status/1335373029245775872?lang=en" rel="nofollow">https://twitter.com/catfish_man/status/1335373029245775872?l...</a>
ECC matters, even on the desktop, it's not even a discussion, to me.<p>If you think it doesn't matter: how do you know? If you don't run with ECC memory, you'll never know if memory was corrupted (and recovered).<p>That blue screen, that sudden reboot, that program crashing. That corrupted picture of your kid.<p>Who knows.<p>I'll tell you, who knows. God damn every sysadmin (or the modern equivalent) can tell you how often they get ECC errors. And at even a small scale you'll encounter them. I have, on servers and even on an SAN Storage controller, for crying out loud.<p>If you care about your data, use ECC memory in your computers.
I have multiple times postponed buying new computers for YEARS, because I'm waiting for intel to get their head out of their ass and actually let me buy something that does ECC for desktop. (incl laptops)<p>I would have bought computers when I "wanted one". Now I buy them when I <i>need</i> one. Because buying a non-ECC computer just feels like buying a defective product.<p>In the last 10 years I would have bought TWICE as many computers if they hadn't segmented their market.<p>Fuck intel. I sense that Linus self-censored himself in this post, and like me is even angrier than the text implies.
Linus is absolutely right.<p>I am trying to get a laptop with dual NVMe (for ZFS) and ECC RAM. I can't get that, at all - even without the other fancy things I would like such as a 4k OLED with pen/touchscreen.<p>In 2020, even the Dell XPS stopped shipping OLED (goodbye dear 7390!)<p>I will gladly give my money to anyone who sells AMD laptop with ECC. Hopefully, it will show there's demand for "high end yet non bulky laptops"
I wish this was more of a cohesive argument. He says he thinks it’s important and points to row-hammer problems but doesn’t explain why. Probably because the audience it was written for already knows the arguments of why, but this is not the best argument.<p>If in doubt, get ECC. Do your own research on how it works and why. This post won’t explain it, just will blame Intel (probably rightfully so).
A relevant Bryan Cantrill talk segment on this, which heightens the paranoia around this. Namely, firmware hiding correctable errors and only reporting uncorrectable errors.<p><a href="https://www.youtube.com/watch?t=2104&v=fE2KDzZaxvE" rel="nofollow">https://www.youtube.com/watch?t=2104&v=fE2KDzZaxvE</a>
Consumer awareness about ECC needs to be better, with recent security implications I simply can't understand why more motherboard manufacturers don't support it on AMD. Intel of course is all to blame on the blue side, I stopped buying their overpriced Xeons because of this.
I'm about to write some code that will allocate a random buffer, take a checksum of it, and just sit on the buffer, periodically checksuming it again until a bit flips. Or maybe even allocate a buffer of zeros and wait until a non-zero appears in it.
Does anyone know why ECC memory requires the CPU to support it?<p>Naively, I can understand why error <i>reporting</i> has dependencies on other parts of the system, but it would seem possible for error <i>correction</i> to work transparently.
A couple of years ago there was advancements that claimed to make Rowhammer work on ECC RAM even with DDR4 [1]. Is that no longer a concern for some reason?<p>I would think the only guaranteed solutions to Rowhammer are actually cryptographic digests and/or guard pages.<p>[1] <a href="https://www.zdnet.com/article/rowhammer-attacks-can-now-bypass-ecc-memory-protections/" rel="nofollow">https://www.zdnet.com/article/rowhammer-attacks-can-now-bypa...</a>
ECC is required on mission critical hardware.<p>I have spent 36 years fielding embedded devices in core network (D1/E1, SONET, ROADM/MPLS, Cellular basestation) and I will tell you that large ECC covered memory arrays always show small numbers of correctable error events over the course of a year. I have seen, over the course of my career, exactly one controller card replaced early in the field, because it started throwing excessive recoverable ECC events over time, until it hit a threshold of 10x the average of a typical board. On the order of ten recoverable ECC events per month instead of one event per month. I have never observed a logged non-correctable ECC event in the field. In the lab, yes, but never in fielded equipment.<p>If you are fine with your PC experiencing one or two bits flipped in memory every month, then you really don't need ECC. That is the question you need to answer.<p>For mission critical systems? ECC is a requirement.
> ECC availability matters a lot - exactly because Intel has been instrumental in killing the whole ECC industry with it's horribly bad market segmentation.<p>The phrase that strikes me is "horribly bad market segmentation". I agree 100%.<p>Remember when the Pentium/pro/2/3 could operate in single and dual socket configurations with ECC? The same CPU that plugged into your low end consumer board could also plug into a high end server/workstation board. All you needed was the right motherboard.
Please someone correct me if I'm wrong, but as far as I can remember memory with extra capacity for error detection used to be a rather common thing on early PCs. That really only changed a couple of decades in, in order to be able to offer lower prices to home users who didn't know or care about the difference. Probably about the time, or earlier, when with some hard disk manufacturers megabytes suddenly shrunk to 10^6 bytes (before kibibytes or mebibytes where a thing, btw).
Over the years, I don't think I've ever been able to explain to anyone that their memory error could have been caused a cosmic ray without being laughed at.
In case the page os not loading, refer to the wayback machine[1] for a copy<p>[1] <a href="https://web.archive.org/web/*/https://www.realworldtech.com/forum/?threadid=198497&curpostid=198647" rel="nofollow">https://web.archive.org/web/*/https://www.realworldtech.com/...</a>
For the average user, what’s the impact of bit flips in memory in practical terms?<p>I am not talking about servers dealing with critical data.<p>Suppose that I maintain a repository (documents, audio and video), one copy in a ZFS-ECC system and one in an ext4-nonECC system.<p>Would I notice a difference between these two copies after 5-10 years?<p>That tells us if ECC matters for most people.
My favorite example is a bit flip altering election results:<p><a href="https://www.wnycstudios.org/podcasts/radiolab/articles/bit-flip" rel="nofollow">https://www.wnycstudios.org/podcasts/radiolab/articles/bit-f...</a>
When I chose my PC parts when Ryzen first came out I tried to get ECC parts. The RAM was obtainable, the problem was that no motherboards had ECC support at the time. I hope the situation has improved by the time I get my next motherboard/cpu upgrade.
For those out of the loop like me, ECC does indeed stand for error correcting code. <a href="https://en.m.wikipedia.org/wiki/ECC_memory" rel="nofollow">https://en.m.wikipedia.org/wiki/ECC_memory</a>
I always have that conversation when ZFS comes up. Some peoples think ZFS NEEDS ECC, but in fact ZFS needs ECC much as every single one FS in Linux. And every single reliable Machine needs ECC.
I have:
Asus PRIME A520M-K Motherboard
2x M391A2K43DB1-CVF (Samsung 16GiB ECC Unbuffered RAM)
AMD Ryzen 5 3600<p>I specifically was looking for bang for buck, low(er) wattage and ECC.
Fantastic burn by Linus Torvalds whom also had some skin in the CPU game.<p>Offtopic, I wonder if he trawls that site regularly. And eventually I wonder, is he here also? :)
I was reading it and thought: wow, this guy is absolutely right! Great things he's talking about. After reading it, i saw it was Linux Torvalds :D
I browsed some online listings for ECC memory modules, and they seem to be sold one module at a time. Standard DDR4 modules are sold in pairs, to benefit from dual channel mode.<p>Does ECC memory support dual channel??
It really does. I did a write-up recently on it as I was diving in and understanding the benefits: <a href="https://qvault.io/2020/09/17/very-basic-intro-to-elliptic-curve-cryptography/" rel="nofollow">https://qvault.io/2020/09/17/very-basic-intro-to-elliptic-cu...</a>
Linux is accusing Intel of killing ECC intentionally. But that is not really the case, they just wanted people to pay up.<p>If you care about ECC, you pay for Xeon. Majority of consumers don't run critical applications on their devices, so they are happy with a cheap device that may crash once in a while.<p>AMD is only changing the game because they are trying to undercut Intel. They have been putting pro features into all of their CPUs including over-clocking, extra PCIE lanes and ECC.<p>Honestly, what is the point of bullet-proof hardware when the software reliability (at least on consumer devices) has gone down to two nines.
ECC memory is predominantly used in servers where failure absolutely must be identified and logged. The desktop market to a lesser extent due to lack of mission critical tasks being run from there.