A negative leap second (the appalling shitshow from the article) would make us skip a second, though.
Going back one second is what generally happens now when a Linux box does the "inserting leap second" thing. It goes from 23:59:59.999999 to 23:59:59.000000, then runs that whole second again. You get to 23:59:59.999999 again, and then you finally roll over to 00:00:00.000000.
From the perspective of the typical time_t rendering of Unix time, there is no way to uniquely represent that 61st second. It just "disappears".
Having lived through systems going backwards some 17 seconds due to a botched NTP-GPS appliance config, I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid.
This didn't have to happen. They could have used monotonic from the get-go, and then they wouldn't have died when the clocks got yanked backwards 17 seconds to the proper time base.
>'From the perspective of the typical time_t rendering of Unix time, there is no way to uniquely represent that 61st second. It just "disappears".'
This is a great and really intuitive summary of the problem. Is there a certain class of problem related to disappearing second? Like are these more likely to be filesystem issues or things that rely on timestamps? Or are there second order problems as well?
>"I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid. I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid."
Sorry if this is a silly question but was that check simply that "time t1 is greater than time t0"? Also was the duration of that outage(17 seconds) or would this have been equally catastrophic at a single second?
Well, okay, so, the problem is like this: let's say you wanted to schedule something to happen during that exact extra second in the end of June 2015 when we were all standing around watching UTC do its little extra dance. You pick 1435708800. Trouble is, that Unix time applies to both 23:59:59Z and 23:59:60Z on that particular day.
You can't target it beforehand or after. It's just... gone.
It's not a problem from the point of view of programs, since they just got whatever time_t value they got, and they don't know the bigger perspective. It's more of a mapping from outside->in problem.
Put it another way: try writing a program that'll call clock_gettime() and will say a message at a later time you select. You can't put in 23:59:60Z because there's no way to represent it, and indeed, you won't even be able to tell when the time comes unless you special-case it and notice _that particular second_ repeating itself... or reach into the kernel to look at the leap bit, or worse. It's a real time in meatspace, but you can't target it with the tools at hand. That's the problem.
Regarding the 17 second thing, that's because someone decided to switch off the thing which (correctly) applies the adjustment factor to GPS time to make it NTP time. There was a 17 second difference at the time (GPS to NTP), and with it off, we were shipping GPS time to hosts as if it was NTP time.
In theory, any regression of the clock long enough to not let the actual passing of time push it past the sanity check time point in the lock stuff would have caused this. The thing is, a small-scale time step (from ntpd, say) normally happens at boot up, not later, and it's on a system by system basis.
The 17 second excursion happened on hundreds of thousands of machines all at once, and, yeah, it was noticed.
> Having lived through systems going backwards some 17 seconds due to a botched NTP-GPS appliance config
Well then, it really was botched. I was running an off-line hodge podge of 50 or so RH7.3 + Win2K systems back in 2002, where I had to manually fix the few second drift every couple of months, and the RH machines all slewed faster/slower to adapt, non ever went backward or jumped forward (Don't remember how the Win2K handled it; They weren't mission critical and I didn't care much).
Distributed alogrithms often depend on precise clocks to maintain consistency. See google spanner for instance.
Updating the clock one second means taking a whole cluster offline untill you are sure the clocks are synchronized exactly before bringing them online again. It brings an outage and potential of headaches.
If the clocks are not synchronized you break the consensus, might cause transactions to commit partially or cause inconsistent reads and writes.
Any distributed system that breaks in the presence of leap seconds is necessarily poorly designed.
The fact that any non-UI code uses any time standard except TAI is a very unfortunate historical mistake that will hopefully be cleaned up over the next few years.
Thankfully since 2013, Linux kernel has supported TAI.
> The fact that any non-UI code uses any time standard except TAI is a very unfortunate historical mistake that will hopefully be cleaned up over the next few years.
Unfortunately, almost everything is UI code in some way, so this won't be cleaned up.
The problem with TAI is that you cannot convert a future time to UTC if it is farther than 6 months or so in the future (because leap seconds have not been decided), so unless it never needs any conversion to/from UTC, you can't use it. (are logs UI? is crontab UI? eventually everything is)
Furthermore, you can't build an embedded no-need-to-update-every-6-month system with available common protocol - you would need to continuously feed it the list of leap-seconds, for which AFAIK there is no widely deployed protocol.
But if you use UTC, there are many protocols (TIME, NTP, SNTP, PTP), at least one of which you'll use anyway because of clock drifts, that will let you keep working.
> The problem with TAI is that you cannot convert a future time to UTC if it is farther than 6 months or so in the future [...]
Centimeter's argument is that such time should not be converted to TAI in advance, much like the local time for the future point (since the local time zone offset may change). All points on this also apply to the local time, e.g. it's already true that non-updatable embedded systems can't reliably do the local time calculation.
They can’t do local time, but they can do UTC (and do it) without much investment if you run an NTP client. (Have 500 or so such units deployed, SNTP was giving too much trouble, switched to chrony at some point and those units require much less support)
Pretty sure the Spanner case isn't using either one. Google's systems futz with the speed of time near the leap second to distribute it across a wide span about midnight, UTC.
Linux had two different kernel bugs around this, IIRC. MySQL got confused too (although that one just ate cpu until you restarted the daemon, it didn't crash, and it didn't prevent service)
Going back one second is what generally happens now when a Linux box does the "inserting leap second" thing. It goes from 23:59:59.999999 to 23:59:59.000000, then runs that whole second again. You get to 23:59:59.999999 again, and then you finally roll over to 00:00:00.000000.
From the perspective of the typical time_t rendering of Unix time, there is no way to uniquely represent that 61st second. It just "disappears".
Having lived through systems going backwards some 17 seconds due to a botched NTP-GPS appliance config, I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid.
This didn't have to happen. They could have used monotonic from the get-go, and then they wouldn't have died when the clocks got yanked backwards 17 seconds to the proper time base.