[NeutronBoy]

> My idea was a dedicated, battery-backed piece of hardware that reliably stored time plus could sync other machines. Plugs into an interconnect with ultra-low latency. One for each datacenter.

Google have a variant on this, where they use a GPS receiver in each data centre to provide an accurate time source for local machines.

[dingo_bat]

I've seen this being used extensively in sensor control centers (power grid/power plants), so it's definitely not limited to Google.

Basically by accurate and precise timing, they are able to reconstruct and pinpoint the origin of a failure.

[nickpsecurity]

No, they use a GPS with 7ms time spread for servers as you said. What Im describing is a custom device set against an atomic clock to nano or microsecond accuracy which then does the same to the servers via low latency interconnects. Optionally with time server & dedicated networks for reduced admin overhead.

Should do a lot better than 7ms with performance implications.

[packetslave]

It's actually a mix of GPS and atomic clocks for diversity. See the Spanner paper section on TrueTime for details