[busymom0]

> Silent data corruption (SDC) is the most expensive, where a faulty GPU produces wrong answers without crashing anything, which means a multi-day training run can complete normally and the resulting model weights are quietly poisoned.

Anyone know how these get caught ultimately?

[wmf]

Usually silent data corruption isn't caught. If a file fails to open you might realize it's been corrupted.