[Doohickey-d]

Im curious about this: because in my experience (working on smaller services though), a small number of errors is always there, as a "baseline".

Recently there was this: https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips"

Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on a network, there's potential for a lot more random errors.

[KPGv2]

Bitflips are something that can happen in consumer-grade RAM, so that tracks (and it's comforting that wayward cosmic rays are a substantial reason for an application's crashes!), but on enterprise servers, they will run ECC RAM that is very resistant to bit flips.

This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Because bit flips, for all intents and purposes, cannot happen. The RAM itself detects and corrects for them.

I wouldn't expect bit flips to be a significant contributor to enterprise problems.

[Anon1096]

Bitflips specifically may not be; things like network issues, noisy neighbors, row/rack/host maintenance (leading to a downed and migrated host) absolutely are things that happen at high frequency at scale and cause your background level of errors to be more than 0.

[maccard]

You've completely missed the point - It's not about bitflips it's about errors that are outside the scope of what's fixable.

[KPGv2]

I suppose I misunderstood what the "random error" was supposed to mean. I wouldn't call a network error a "random error" because it's caused by things that are internal to the system (entities using a network). A bit flip is caused by an external factor: cosmic radiation. To me, that's what a "random" error is.

If your network goes down because of a DDOS, or part of your system overheating, that's an internal issue you had control over.

If a bit flips because of cosmic radiation, you can't really do anything about that, and it's utterly unpredictable. That's "random" to me.

[bobthepanda]

It’s where monitoring for 9s is more important at that scale than absolute errors. So long as degradation is graceful or retried it should not be a massive problem.

It does require constant tuning and adjustment though.