Wow, that's a pretty bad illustration of leap smearing. I think a much nicer illustration would have been roughly like this (with obvious limitations caused by the ASCII spacing):
Real: true time, showing the 61st second
Comp: computer time, only counting 60 seconds
not smeared,
time counted at regular speed
...--->| |<---...
Real 54 55 56 57 58 59 60 00 01 02 03
Comp 54 55 56 57 58 59 00 01 02 03
|<- smearing ----------->|
time counting slowed down
by 20%
Also, who the heck decided to call it a "leap smear"? There isn't anything "leap"ing about it anymore. If you are changing from "leap second" to something indicating smearing, "smear second" is the logical choice.
Why aren't leap seconds treated as a time zone change, like a daylight saving time transition? Keep the underlying clock counting the number of actual seconds since midnight, January 1st, 1970 (or whatever your epoch of choice is) and just add one more second into the offset to local time. Is there some big problem with this approach I haven't thought of, or is there just too much history with doing it the other way to change now?
I don't understand why either. Counting up from 1/1/1970 and then making adjustments for rendering only seems like a simple, reliable, and time tested solution.
I guess people don't want a database of leap seconds and a lookup each time time is rendered, but ultimately it seems like the best solution which is least likely to break things.
Yeah. I'd like to see us all set our hardware clocks to TAI and everything else comes from the time zone files. It won't solve everything because lots of protocols have broken notions of time, but at least we could all agree on what the hardware time is supposed to mean and there would be no worry about leap seconds at that level.
If your dates and times and locations vary enough, you already can't do a mod 60 check on UTC to detect a minute boundary, as various historical time zones include seconds-level offsets from UTC. For example, US cities were on local solar time until the latter part of the 19th century, and the changeover to uniform time zones involved lots of seconds-level offsets. Interesting stories here:
It appears to have this stuff down to the second. For example:
# Zone NAME GMTOFF RULES FORMAT [UNTIL]
Zone America/New_York -4:56:02 - LMT 1883 Nov 18 12:03:58
# Zone NAME GMTOFF RULES FORMAT [UNTIL]
Zone America/Chicago -5:50:36 - LMT 1883 Nov 18 12:09:24
Although this won't really be accurate, because it doesn't have every single city. The New_York zone covers the whole eastern time zone even though all the cities in that zone would have been different before 1883. But it is in the database, even if it's not quite right for anything outside of the named cities.
Many (most?) important computer systems use UTC precisely to avoid time changes like the DST transition. Just one example that comes to mind is that DST can cause important cron jobs to repeat, or not occur at all.
Right, which is why it seems like the obvious thing to use TAI as the underlying base time in a computer system, and have UTC be just another time zone offset on top of that, so you don't have to deal with leap seconds either.
Exactly. The underlying clock would store actual seconds since 1970, and when you request UTC it would add the current leap second offset. Which is more or less how GPS does it already, for an actual working example.
The leap second caused our databases to all peg at 100% CPU/memory consumption.
I remember being utterly ridiculed that weekend, after swearing about fixing servers on Facebook.
It didn't take long for apologies to arrive on the following Monday, once the corporate sysadmins logged in and saw anything Java-based had also massively, massively failed.
If you experience that problem this year, this supposedly can be fixed without even restarting the process by running "date -s now". I say "supposedly" because I can't validate, but based on my reading of the problem and why that works, I have no reason to believe it won't.
My brother used to work for an atomic clock company. Their array of clocks was the 2nd most precise next to NIST in Colorado, he said, and when small fluctuations in the earth occurred, their's was another point of contact for GPS timekeeping (allegedly). Reading through the OP, I remember that this sort of mid year leap second happened a few years back with the Japanese major earthquake. My brother said that they had to reset all the clocks about a millisecond due to the shifts in the rotation of the earth because of the quake. One would think that the programmers would have written into these market timing devices a simpler way to adjust these calculations, seeing as these timing updates keep on happening.
Clock sync is one of the areas where the markets and their participants actually deserve their reputation for sophistication.
That said, I'd be surprised if keeping the clocks together is what is causing the worry. Much more troubling are third party bugs (kernel, jvm, db, etc) and incorrect assumptions in their code.
This same story can be equally applied to any industry honestly, but the markets make for better headlines.
This sounds rather exaggerated to me. 10% of the systems will have trouble with it? Perhaps during the leap second something will bug or look weird, but outside of that second I doubt it's even 1% that will have trouble recovering. Also, many systems don't even support the leap second, nothing will happen at all until they sync with time servers the next time and correct for the second. Many systems are off a second or so anyway.
As a quick example, Javascript is not designed to handle leap seconds correctly, so I doubt we'll ever see the value 60 out of new Date().getSeconds(). Things will just be off by 1 second for a second.
Edit: Another example from the Windows Time service:
> The Windows Time service does not indicate the value of the Leap Indicator when the Windows Time service receives a packet that includes a leap second. [...] Therefore, after the leap second occurs, the NTP client that is running Windows Time service is one second faster than the actual time. This time difference is resolved at the next time synchronization.
It will just be a second off and resynchronize next time.
If your system is one of those 10% and if your trading algo is executing fucked up trades because of timing issues and you lose real money - I wouldn't call that exaggerated.
Also, not sure if any of the systems mentioned in the article run JS for critical server side code.
High-frequency trading and other important realtime processes that handle money should of course be tested for this scenario (either by simply not trading for a few seconds before and after, or by properly handling it). The vast majority of the computer systems around the world are however not HFT systems and the world will continue to work normally. No Y2K-like doom scenarios needed like the article suggests.
We've been here before, June 30th 2012 also contained a leap second (see Wikipedia). The article only forecasts doom without looking back at previous leap seconds and evaluating its effect.
> We've been here before, June 30th 2012 also contained a leap second (see Wikipedia). The article only forecasts doom without looking back at previous leap seconds and evaluating its effect.
Clearly you didn't read the article.
If you had you would have seen that they noted that last time it was on a weekend when all the markets were closed.
That isn't what you said in your post, now is it? You said "We've been here before", but as the article points out, we have not.
In any case,
Traffic lights and apache for the most part do not care what time it is - and certainly not to the second. Market orders are timed to the millisecond, they care very much.
Slewing vs stepping or any other de-synchronization could cause chaos with them, not so with most other fields. You might not care about HFT, but this applies to all orders, not just HFT. And like it or not the stock market plays a large role in everyday life.
Unless you are someone who keeps all your money in physical cash and drives cars for a living, there is no way HFT is less important than traffic lights; one keeps the economy of the world flowing, the other stops people getting driving tickets.
"Being a second off and resynchronizing next time" could be a million dollar difference, there is no way that you can shrug that off.
As a developer for a financial firm, this leap second is an extremely big deal; the fact that this is the first time in history that financial markets (which are normally millisecond accurate) have had to deal with this phenomenon will absolutely result in something somewhere going wrong.
This article isn't exaggerated; it's there to highlight what's going on for the traders and brokers who aren't technologically advanced.
That leap second pinned loads of Linux boxes at 100% CPU. I'm a customer of Hetzner, they sent out a mail asking the customers to check their machines as their power draw increased by a megawatt in an instant.
Another reason to just have artificial delays in trading. If every exchange had a minimum of, say, a minute after an order is submitted until it is executed (during which time it couldn't be amended or cancelled of course) it seems you would greatly reduce the risk of "flash crashes"and unnatural manipulation of the market.
I know "market liquidity" is an argument for allowing HFT, but haven't hard a good explanation about why the economy would suffer so much if the ultra high frequency trading just wouldn't exist.
Note: I have about as much knowledge about this as people who suggest how SpaceX should improve. Just sayin.
Of course computer systems everywhere will have issues with an extra seconds (deltas being negative that can't be etc) but most of those systems aren't trading systems. Why is it that "markets" are in the focus of this? At y2k we were worried about planes falling out of the sky. Surely there must be worse things than stock market computer systems being confused?
> but haven't hard a good explanation about why the economy would suffer so much if the ultra high frequency trading just wouldn't exist.
The short explanation is that the speed is a tool for managing risk (like most other market tools) and by removing it, you remove that ability. That risk must be priced into the market somewhere and that will be in the spread that everyone pays.
Will that be worse than what we currently have? Who knows, but what we have is working pretty well with regard to providing liquidity (it is cheaper and easier to get in more markets than ever before and the margins are as low as they have ever been on providing it), so why mess with something that has so few down sides?
> Surely there must be worse things than stock market computer systems being confused?
There are, but Bloomberg doesn't specialize in them, and they don't garner nearly the HN upvotes.
> why mess with something that has so few down sides?
When I look at high frequency trading, it says to me that the game is rigged. It's like going on Jeopardy up against a machine contestant that always buzzes in first.
How on earth can I possibly succeed as an individual investor in a trading environment where institutional investors are so privileged? How many other ways is the ostensibly neutral marketplace overseer profiting by offering those who pay-to-play opportunities to shave pennies, nickels, dimes, and dollars from me?
If investing is not your full-time occupation, you shouldn't be trading every day. (You probably shouldn't be trading every week.) If you expect a regularly-traded security to appreciate over the next month, it's entirely possible to enter an order to purchase that security, and later to sell it, without being beaten out of a few pennies by HFT.
> When I look at high frequency trading, it says to me that the game is rigged.
And when I hear this complaint, I don't understand the rationale. Why shouldn't participants that do something professionally, invest in infrastructure, invest in research or other kinds of IP have an advantage. If they didn't that wouldn't be a market, that would be a lottery.
Quite simply, the market dynamics that make it such that you might want to be involved in it as an individual are created by the sophisticated market participants engaging in individual zero sum trades that add up to a beneficial whole and they have been for hundreds of years.
Computers have made this process more efficient and you should be celebrating your ability to trade for cheaper than ever. That you don't is a sign that you don't understand how the markets work now nor how they have ever worked.
Edit: the more I thought about this, the more I decided it was overly harsh. I think well informed participants can disagree about the values of HFT and needn't celebrate it. I do think that the idea that markets don't (or shouldn't) reward more engaged participants is fundamentally broken.
HFT is a tool of oppression. Who can afford it? Who has the time, capital, and personnel to make and deploy HFT algos? And ultimately, who is really benefitting the most from this technological advance?
Could "High Frequency Trading As A Service" be a thing? Someone provides the infrastructure and a nice API to talk with the market and runs your algorithms on their hyperfast computers at the heart of the stock exchange for a fee.
Every day markets shut down at 4pm except for very limited after hours trading. Every weekend down. 10+ days a year, down for market holidays.
We're talking on the order of zero liquidity for 30% of the days (weekends and market holidays) in a given year, and severely limited liquidity outside of market hours.
If liquidity was as important as we like to say, this just wouldn't be the case.
I'm not sure I understand your argument. Let me see if I can restate it:
"If liquidity was really required at the millisecond level, market participants would not accept markets being down for holidays or weekends."
But that is not the point of the millisecond (or sub) resolution. The point is so that the people that provide the liquidity during operational hours can add as much information as they can into their pricing models, thereby requiring them to take on less risk with each bit of liquidity they provide. This less risk gets passed on as savings to the rest of the market participants in the form of smaller spreads. Any increase in that resolution increases the risk to the market makers, who will therefore need to increase the spread to compensate, making trading for everyone more expensive.
This would be true if the markets were only open 4 hours a week as well, though trends are towards more 24x5 (at least) trading, not less (and I think that is a good thing).
I worked in technical risk management for a large market maker in the US. The latency arms race just drove things towards riskier and riskier practices in our systems.
Soft-realtime systems in C++ instead of safer garbage collected runtimes.
Parsing market data on FPGAs or Cell processors instead of in code written with emphasis on safety.
Distributed trading systems that couldn't afford the latency budget to pass everything through centralized risk management, and therefore not taking into account the entire order book before authorizing a trade. Instead a series of compromises and less optimal failsafes.
It also just lead to more expensive practices--communication between threads using spinlocks instead of the normal waking a waiting thread through a system call; basically converting the spreads we could capture into data center waste heat in order to beat out the next guy.
Artificially slowing things down a bit would have just reduced costs and increased safety for everyone.
That is a bit of an orthogonal issue, but I would say that it doesn't show that the speed of HFT is a problem to the broader market. For instance as a market participant I largely do not care about the implementation of any given market makers risk configs. In fact, even with all the nasty bits in HFT, it is loads better than the system it replaced. There is an open question about if it can be made even better.
I'd also suggest that your experience (many of which I share) do not necessarily mean that speed is making things riskier, only that your centralized risk management was not as good at lowering the cost of risk as opposed to getting fast. As for the dc energy to spread arbitrage game, that is the result of far reaching policy decisions well outside of HFT and I largely agree that energy is too cheap.
I think that any artificial slowing of the market is likely to have really bad unintentional issues. I'd much rather we incentivize the market to fixate on something we care about (ie price) rather than try to subvert the laws of the market.
One approach that appeals to me is to remove the sub-penny rule that is artificially propping up spreads.
> it is cheaper and easier to get in more markets than ever before and the margins are as low as they have ever been on providing it
High enough that the firms are at least receiving a normal profit. And given the opportunity cost that all these smart people incur by working in this industry, this normal profit is rather big, and no doubt that the extraction of this normal profit has non-neglible deleterious effects on the economy.
> If every exchange had a minimum of, say, a minute after an order is submitted until it is executed (during which time it couldn't be amended or cancelled of course) it seems you would greatly reduce the risk of "flash crashes"and unnatural manipulation of the market.
Sorry, but a minute delay is definitely not a solution to your idea of unnatural market manipulation. Artificial delays can be seen in the Chinese equities markets - trading halts given a +/- 10% intraday move, trading halts for up to 10 trading days before news announcements, you can only see quotes every half a second -- all measures to benefit retail traders over institutional traders, a paradigm the American markets inherently oppose. HFTs solve that problem by providing retail investors the same kind of access as institutional investors.
>I know "market liquidity" is an argument for allowing HFT, but haven't hard a good explanation about why the economy would suffer so much if the ultra high frequency trading just wouldn't exist.
HFT is a natural effect of electronic trading in contract to traditional forms of market making and liquidity providing. More often than not, thin spreads are often returned back to retail investors.
Other than some anecdotal evidence of "flash crashes" [1][2], how does HFT negatively impact the market?
This seems to be something that would benefit from standardization and modularity...
Pausing for a second, going back a second in time or changing how long a second is all are hacks that are bound to cause some edge case failure somewhere. I'd like to see a minute have a 60th second just as February has a 29th day.
The problem is that the effect being compensated for is so tiny that it's really noisy. It's not like leap years, where all you have to do is measure the actual period of the Earth's orbit very precisely and come up with a nice way to fit the appropriate correction factor into the calendar.
The Earth's rotation is slowing, but it's not a steady process. For example, the Boxing Day earthquake in 2004 permanently decreased the length of the day by about 3 microseconds. Here is a graph of the changes over the past few decades:
As long as we have a standardized way of determining when leap seconds are going to happen, some authority determining and announcing, there shouldn't be problem. Leap years are predictable, but leap seconds can be accounted for a decade after an event disrupts the pattern.
Also, given all that variance and the fact that we only seem to be shifting around 1 minute per century at the current rate, we probably could get by having a leap minute every few decades when necessary.
This works fine for any system trying to interact with other computers and humans while keeping consensus on the time. Any system that is trying to determine earths yaw would no longer be able to rely on F(time), but that is only a minor inconvenience.
How is that different from what's being done now? There is a standardized way of determining when leap seconds are going to happen (they always happen on June 30th or December 31st when the deviation reaches about 0.6s), there is an authority that determines and announces it (the International Earth Rotation and Reference Systems Service) and it is, in fact, done by causing the last minute on the day in question to have 61 seconds instead of the usual 60.
The complication is just that there are a whole lot of systems out there that assume that every minute has exactly 60 seconds with no exceptions, so you end up with crazy workarounds like smearing to hide that 61st second from such systems.
>How is that different from what's being done now? There is a standardized way of determining when leap seconds are going to happen (they always happen on June 30th or December 31st when the deviation reaches about 0.6s)
It's different because you don't have to keep track of what an authority is saying or deal with the actual time adjustment except once in a century. Once in a century is acceptable since we will only deviate by around a minute at our current rate.
Leap seconds are announced months in advance and can only occur in 2 possible positions in an entire year. It's pretty standardized, especially compared to the mess that is daylight savings time, where changes can be announced only a few days in advanced (e.g., Egypt) and possibly even retroactively.
The problem with leap seconds is that time is often represented internally as seconds from the Unix epoch, and this simple integer counter assumes leap seconds don't exist. In effect, there is no way to actually represent a leap second in many computer systems, so the question of what you do is basically a choice of the least painful lie. Handling leap seconds in the situations where people care about them requires a lot of complexity for an event that happens only once every few years.
There's a movement to abolish leap seconds, which I am very much in favor of.
>It's pretty standardized, especially compared to the mess that is daylight savings time, where changes can be announced only a few days in advanced (e.g., Egypt) and possibly even retroactively.
It clearly isn't since the original post highlights three different ways people adjust the time making all these organizations lack consensus on the time.
It'd benefit from simply dropping it. Why should all of us pay and deal with these stupid hacks, just because some astronomers can't fix their software?
Move to leap hours, and by the time it's needed, let the AIs sort it out.
Leap hours are probably acceptable, but at the point that you're only deviating by an hour every few millenia, can't it be safely accepted that the time has a different meaning and between generations it be accepted that 8PM means night time in the summer rather than 9PM? I doubt that we will even be using the same system by then anyways.
Can someone please explain why this keeps being explained as 'earth's rotation is slowing down'?
Having a scheduled addition of the leap second only corrects an issue resulting from our timing being slightly off (by about 1 second every couple decades). Now, if the second had to be add in changing frequency... the changing in frequency would be a result of the earth's rotation changing... but even still that wouldn't be why the second itself is being added, is it?
"One second every couple of decades" grossly, grossly overestimates our timing inaccuracies. Multiple orders of magnitude.
The problem is in some sense, the opposite... our timing is now so good that we can notice the natural variations in Earth's rotation to astonishing accuracy. Thus, if we want to keep the clock in sync with what the Earth is actually doing, we need to add the leap second.
(There's quite a bit of debate about that "if", and IMHO right now industry after industry seems to be experiencing major, expensive disruptions and repeatedly incurring risk, to save astronomers from having to consult a lookup table that they of course would build into computer programs and pretty much forget about ever after. But I digress.)
The Earth's rotation really does wobble by some amount due to natural shifts in density, temperature changes, and any number of other things. It's why we don't have a schedule of leap seconds for the future. If it were merely a matter of accurate timing we'd have a schedule, because our timing is accurate enough by orders of magnitude to make a schedule for a hypothetical perfectly-steady or perfectly-slowing Earth for millennia in advance. The problem is that we can't predict what the Earth is going to do.
Astronomers use sidereal time instead of solar time anyway, so they're used to making time corrections, the sidereal day is only 23 hrs, 56m, 4 seconds and change. Equivalent to the length of a day if the light from the sun was coming from an infinitely distant source. That way at a given sidereal time, anything far enough away will always be in the same place in the sky.
The change in rotation is not consistent.
Earthquakes change the earth's rotational inertia by raising and lowering land near the equator. Sometimes they jog the entire planet forwards or backwards in its rotation by a slight increment. Tidal interactions vary significantly enough to have an effect.
Even man made changes are measurable - the three gorges dam in China moved enough water uphill to have a measurable effect.
And simply put, the leap seconds are added so that clock noon, solar noon don't drift too far apart.
In the meantime, the Paris-based International Earth Rotation and Reference Systems Service will keep track of the gradual slowing of the planet, caused in part by drag created by the Moon. Millions of years ago days were 22 hours long.
I know the speed of rotation changes slowly. But that would be a reason for lengthening a day, not adding a leap second. The leap second is added because our concept of time is slightly off, but it keeps being presented as if we had to add an extra second because of the slowing of the rotation. If that was the case, it would be like those 2 hours that were added in the last few million years. We don't consider them leap years because they aren't a correction but a redefinition (ignore for a moment that humans weren't around to define as a day as 22 hours back then).
Consider the leap year for a second. We add it because our years have 365 days is slightly incorrect (by about .25 days a year). It isn't that the number of rotations per revolution is or isn't changing. That isn't relevant to our use of a 1 day correction every 4 years. If some force caused us to revolve around the sun slightly faster and it took 365 exact, then we could remove the leap year.
Removing the leap year would be a result of our revolution being slightly faster. But the leap year itself is not.
In short, the delta of the leap correction is a result of the change in the system being monitored, but the leap correction itself is because our system of measurement is off.
You can see how a 365.248-day year would be inconvenient, right? A 86400.002-second day would be more inconvenient, even though the point of TFA is that a 86401-second day is also somewhat inconvenient.
Seconds are not defined by the rotation of the earth, any more than they're defined by its revolution.
I imagine when the original SI second was specified, there was no need for leap seconds, and if the Earth's rotation had remained exactly the same since then then we would never have needed them?
I would suggest the whole thing can be avoided by the exchanges setting their own ticks. Any and all Bid/offers are queued during the tick and then applied, in order, during the next tick.
before I did some research I thought this would be a good "anti-HFT abuse" idea but I still think it has considerable merits.
The article states that the planet's speed of rotation is slowing down, albeit very slowly. Is the implication that at some point in the (distant) future the rotation will stop completely, or will it just reach a new equilibrium? Could it in fact speed up again?
Our sun will turn into a red giant and destroy earth long before earth has a chance to stop its rotation, as we're talking about an increase in milliseconds per century. Since 1820 the earth's rotation slowed down by about 2.5 milliseconds.
The Earth's rotation speed can be influenced by many things and can both be sped up and slowed down. For example, the Sumatran earthquake in 2004 shortened the rotation period by 3 milliseconds. In theory negative leap seconds could be added but none have yet.
This gives rise to problems for computers since they are irregular and unpredictable, and can't be made part of algorithms. In extent, to properly calculate a precise time difference (actual elapsed time) over several years, you actually need to consult a table of added, historic leap seconds and take these into account too...
Simple. What's happening is that in the past we were measuring seconds relative to the duration of a day, but since then we've switched to atomic clocks.
So the day is not 24 hours or 86400 seconds, but rather 86400.002 seconds on average. That's about 2 milliseconds or more of deviation per day and for a whole year that's about 0.7 or 0.9 secs worth of deviation. Yet we still pretend that the day has precisely 24 hours, hence the need for leap seconds.
Given that since 40 years ago since leap seconds were adopted about 25 leap seconds have been scheduled, that sounds about right.
26 seconds over 43 years (1972-2015) is 0.60 seconds/year. If this matches the average offset from 1820...1972 (which would add up to about 92 seconds), it means that our average is wrong by ~0.6 seconds, but the time length doesn't gain additional seconds due to "spinning down" (caused by tidal interaction with the moon, or whatever other effects there might be).
That's actually also what the graph suggests: It's havnig a short-term daviation of +/- 1ms over the course of (judging by eye) weeks, and a long-term deviation of +/- 2ms over the course of decades. The average deviation of 0.6seconds/year (number of leap seconds inserted) translates to an average of 1.65ms/day from 1972-now.
It can't speed up without an external force and at current slow rate it'll take about 2 trillion years to stop (this assumes a linear rate of slowing over time, which won't happen).
Much of the slowing is caused by tidal friction with the Moon (which recedes slightly each year), iirc eventually the Earth will become tidally locked with the Sun (same side facing all the time).
Right. In theory, given enough time, both the Moon and the Earth should end up tidally locked to each other. Currently, only the Moon is locked to the Earth.
In practice, it would take so long for the Earth to tidal-lock itself to the Moon, it's most likely some other major event will happen first (like the Sun expanding and swallowing the Earth).
You could add an extra second by having the previous 1,000 seconds (16 hours) last 1,001ms each, say. Then the scope for confusion is more likely to be limited to software that reads human time, and counts milliseconds, and requires the two to match over time. Probably a much smaller percentage of all programs.
When you know what the problems are likely to be, you can of course pick the approach that's least likely to cause you problems! But if you're going in blind, then this seems like it could be a safer way to do it.
(You might use some kind of a curve to do this, so it's smooth like. I imagine this is what the comment about weighting refers to.)
So the problem is that seconds were measured relative to the duration of a day, but that standard changed and now seconds are measured with atomic clocks, making a day to not be equal to 24 hours, yet we pretend that it is, hence the need for leap seconds.
I understand that this isn't ideal for people dealing with actual dates and times, but the upside is that timespans expressed in seconds are now universal, so for example Unix timestamps actually have meaning that doesn't change according to earth's rotation.
