[evil-olive]

> most distros end up running SystemD with timesyncd instead

which tends to be "good enough" for the average desktop user, but anything even vaguely server-ish should run a full NTP implementation such as chrony and not an SNTP one such as systemd-timesyncd.

a few years ago at $dayjob we had a fleet of CoreOS hosts. CoreOS, at the time, defaulted to systemd-timesyncd using pool.ntp.org addresses.

our CoreOS hosts, obviously, ran Docker containers.

systemd has a neat "feature" where if your network configuration changes, it'll trigger a time synchronization through timesyncd.

when a new Docker container was started, this counted as a "network config change" and caused a time synchronization.

by itself, this isn't too bad. it caused time syncs to happen more often than they need to, strictly speaking, but shouldn't have caused any further problems.

except...enter "falsetickers". hosts in the NTP pool are run by volunteers. an individual host in the pool may have the incorrect time.

the infrastructure for the NTP pool has monitoring for this, and will kick a host out of the DNS rotation if it's wrong. except this won't happen immediately - there'll always be some lag between when the host starts being wrong and when the monitoring system kicks it out.

and if your hosts are synchronizing their time more often than necessary, it increases the chance they'll do a time sync in one of these small windows where a falseticker is being advertised by the pool.

a full NTP implementation is specifically designed to handle this, of course. a client polls multiple servers, and will discard significant outliers.

SNTP? not so much. I haven't looked at timesyncd to see if it's improved since then, but at the time it would pick one of the [0-3].pool.ntp.org hosts at random, send it one NTP packet, and then jump the time to that response.

...and that's the story of how some of my company's production hosts would have their system time autonomously jump to be 5-10 minutes fast, maintain that time for several minutes to an hour, and then jump back to the correct time, all without human intervention.

[bombcar]

I cheat and have an authoritative NTP server locally and then override dns for pool.ntp.org and friends.

Then at least if I’m off we’re all off together.

[throw0101a]

> I cheat and have an authoritative NTP server locally and then override dns for pool.ntp.org and friends.

Generally if you've made the effort to have internal recursive DNS server(s) for your network, then just enable NTPd or chrony as well and have a single source of Time Truth for your network.

Point to ≥4 NTP servers, even using pool.ntp.org, and you probably don't have to worry about false ticker(s) either.

[sneak]

For bonus points, hook up a GPS with a PPS output to the local one so it's stratum 1.

[willis936]

I do this, with all the trimmings (running in kernel space, PTP simulation, etc). I appreciate that a good estimation of the time inside the non-deterministic OS is being made, but I haven't quite wrapped my head around what it means to extract the time from that non-deterministic OS.

How big is that unmeasured error?

[karlshea]

I do this at home with a Pi, it was a fun project.

[sneak]

SNTP is something else (a text based protocol IIRC) than ntpdate-style oneshot point in time sync, which, while still vulnerable to hitting one server and getting the wrong time, uses the ntp (binary IIRC) protocol to do so.