[loph]

Only Boulder servers lost sync.

To say NIST was off is clickbait hyperbole.

This page: https://tf.nist.gov/tf-cgi/servers.cgi shows that NIST has > 16 NTP servers on IPv4, of those, 5 are in Boulder and were affected by the power failure. The rest were fine.

However, most entities should not be using these top-level servers anyway, so this should have been a problem for exactly nobody.

IMHO, most applications should use pool.ntp.org

[crazydoggers]

I believe if you use time.nist.gov it round robins dns requests, so there’s a chance you’d have connected to the Boulder server. So for some people they would have experienced NIST 5 μs off.

[NetMageSCW]

Who does use those top-level servers? Aren’t some of them propagating the error or are all secondary level servers configured to use dispersed top-level servers? And how do they decide who is right when they don’t match?

Is pool.ntp.org dispersed across possible interference and error correlation?

[mcpherrinm]

You can look at who the "Stratum 2" servers are, in the NTP.org pool and otherwise. Those are servers who sync from Stratum 1, like NIST.

Anyone can join the NTP.org pool so it's hard to make blanket statements about it. I believe there's some monitoring of servers in the pool but I don't know the details.

For example, Ubuntu systems point to their Stratum 2 timeservers by default, and I'd have to imagine that NIST is probably one of their upstreams.

An NTP server usually has multiple upstream sources and can steer its clock to minimize the error across multiple servers, as well as detecting misbehaving servers and reject them ("Falseticker"). Different NTP server implementations might do this a bit differently.

[neonnomad]

I've been running servers for the pool for years. They are checked regularly for accuracy/uptime or it's score goes down in the pool and eventually gets removed. I sync from 5 stratum 1 servers and use chrony these days.

Facebook had a really interesting engineering blog about building their own timeservers: https://engineering.fb.com/2020/03/18/production-engineering...

Really well written for anyone who is interested.

[yardstick]

From my own experience managing large numbers of routers, and troubleshooting issues, I will never use pool.ntp.org again. I’ve seen unresponsive servers as well as incorrect time by hours or days. It’s pure luck to get a good result.

Instead I’ll stick to a major operator like Google/Microsoft/Apple, which have NTP systems designed to handle the scale of all the devices they sell, and are well maintained.