Hacker News new | ask | show | jobs
by ignoramous 2327 days ago
In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used [0], but even enterprise-grade hardware would at times (at that scale [1]) drop and/or corrupt a few bits here and there.

That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly [2].

Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good [3].

[0] https://danluu.com/fsyncgate/

[1] https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...

[2] https://youtube.com/watch?v=swQbA4zub20&t=46m02s

[3] https://youtube.com/watch?v=fE2KDzZaxvE