Hacker News new | ask | show | jobs
by m3047 860 days ago
Ran into a case where a whole datacenter became untethered from its NTP upstream and drifted off into a timezone of its own creation. Customer was failing authentication for a data product we sold them (TSIG was failing). I was on the phone with them for an hour, reassuring them constantly that everything was working for our other customers, tailing logs, and reporting what I saw.

More datacenter stakeholders kept joining the call, most of whom had nothing to do with our data product. Many times I heard people ask "have they found the problem yet" as though.. what? We were the best tech support they had for an entire data center going dark? After an hour somebody noticed that the clocks on servers in the datacenter didn't match up with their laptop; shortly after that I was able to extricate myself from the call... still watching the logs, their downloads started working again a short while later.
1 comments
yjftsjthsd-h 860 days ago
> Many times I heard people ask "have they found the problem yet" as though.. what? We were the best tech support they had for an entire data center going dark?

Possible. Some companies are mostly lacking in competent technical people, so anyone who knows what they're doing will quickly find themselves pulled into every possible task; I see no reason why this shouldn't include external parties.

link
tetha 860 days ago
Especially if things enter the very strange territory, like NTP running wild.

I pretty much remember some time ago, one of our customers had trouble with our on-prem installation. Eventually it seemed that the database had been corrupted. At that point I could tell the poor guy on the other side was running what I was doing with my colleague on a couple of other, similar systems, but I noted he was getting pretty nervous so I figured as long as the clock runs, whatever. It's kinda what I do a lot at work and I don't like letting people in the rain like that.

And eventually we could confirm that large amounts of VMDKs had been corrupted in various ways. Seemed like another vendor hat let the SAN they were managing run full or into some other catastrophic situation. And their backup appliance also didn't work.

link
m3047 859 days ago
> Possible.

I had dealt with these people for several years. In my company, I changed teams several times.

But I still got emails inviting me to internal meetings: they thought? assumed? hoped? ...that I was an internal consultant.

This was pursuant to a larger project and I was on the call for that one and pointed out that I was no longer on the implementation team but was thrilled to be there... because I was. That final / initial deployment resulted in WTFs and I said "woot!" and dropped the call.

link
m3047 859 days ago
> resulted in WTFs

It was a visibility / cybersecurity product and it pretty much immediately paid for itself. ;-)

link