Hacker News new | ask | show | jobs
by tetha 860 days ago
Especially if things enter the very strange territory, like NTP running wild.

I pretty much remember some time ago, one of our customers had trouble with our on-prem installation. Eventually it seemed that the database had been corrupted. At that point I could tell the poor guy on the other side was running what I was doing with my colleague on a couple of other, similar systems, but I noted he was getting pretty nervous so I figured as long as the clock runs, whatever. It's kinda what I do a lot at work and I don't like letting people in the rain like that.

And eventually we could confirm that large amounts of VMDKs had been corrupted in various ways. Seemed like another vendor hat let the SAN they were managing run full or into some other catastrophic situation. And their backup appliance also didn't work.