[michaelt]

A few years back I was interested in microsecond-precise timekeeping.

My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive.

And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch.

I wonder how Facebook are connecting their precise new time servers to their regular servers?

[MisterTea]

This gear will get cheaper soon.

There is a big push in the Industrial automation world to get TSN/PTP deployed more widely and perhaps universally. The idea is to eliminate the dichotomy of real time vs non real time Ethernet networks in control systems from the plant network down to the machine level.

This way you can send EtherCAT frames at equidistant intervals to servo drives in the microsecond range while casually streaming SPC data or Netflix over the same wire.

This is also in conjunction with the newer single pair Ethernet tech that the Automotive world is also interested in.

[throw0101a]

> single pair Ethernet

For anyone wondering:

> In addition to the more computer-oriented two and four-pair variants, the 100BASE-T1 and 1000BASE-T1 single-pair Ethernet PHYs are intended for automotive applications[17] or as optional data channels in other interconnect applications.[18] The single pair operates at full duplex and has a maximum reach of 15 m or 49 ft (100BASE-T1, 1000BASE-T1 link segment type A) or up to 40 m or 130 ft (1000BASE-T1 link segment type B) with up to four in-line connectors. Both PHYs require a balanced twisted pair with an impedance of 100 Ω. The cable must be capable of transmitting 600 MHz for 1000BASE-T1 and 66 MHz for 100BASE-T1.

> Similar to PoE, Power over Data Lines (PoDL) can provide up to 50 W to a device.[19]

* https://en.wikipedia.org/wiki/Ethernet_over_twisted_pair#Sin...

> The IEEE 802.3bu-2016[12] amendment introduced single-pair Power over Data Lines (PoDL) for the single-pair Ethernet standards 100BASE-T1 and 1000BASE-T1 intended for automotive and industrial applications.[13] On the two-pair or four-pair standards, power is transmitted only between pairs, so that within each pair there is no voltage present other than that representing the transmitted data. With single-pair Ethernet, power is transmitted in parallel to the data. PoDL defines 10 power classes, ranging from .5 to 50 W (at PD).

* https://en.wikipedia.org/wiki/Power_over_Ethernet#PoDL

100BASE-T1 is IEEE 802.3bw-2015, 1000BASE-T1 is IEEE 802.3bp-2016. 2.5 Gb/s, 5 Gb/s, and 10 Gb/s over a single pair is 802.3ch-2020: the focus of these is in the embedded automotive space.

* https://blog.siemon.com/standards/ieee-p802-3ch-multi-gig-au...

[throw0101a]

Addendum: The .3cz task force is working on 25/50/100 Gb/s over single twisted pair (two conductors) at lengths up to 11m (point-to-point):

* https://ieee802.org/3/cz/index.html

[abyagowi]

Most (if not all) our switches in the datacenter are capable to perform as transparent clocks. With transparent clocks you can keep the time sync error down to less than 5ns per hop

[mlichvar]

Do the transparent clocks in your datacenter support unicast PTP?

There is a possibility to use PTP as a transport for NTP to take advantage of PTP-specific hardware timestamping. It could also process the correction field, but it seems the switches typically don't support unicast PTP.

The large asymmetry and banding of NTP in the test with Calnex Sentinel suggests it doesn't support the interleaved mode. NTP with hardware timestamping should normally be much more stable and symmetric, closer to PTP.

[throw0101a]

> but it seems the switches typically don't support unicast PTP.

Doing a quick search, the documentation for Cisco, Arista, and Juniper all mention unicast PTP, so it may be the feature is becoming more prevalent.

See also "Enterprise Profile for the Precision Time Protocol With Mixed Multicast and Unicast Messages":

* https://datatracker.ietf.org/doc/html/draft-ietf-tictoc-ptp-...

[mlichvar]

The unicast PTP support can be limited to boundary clocks and the enterprise profile doesn't require transparent clocks to support the unicast mode.

Also, there are different types of PTP transparent clocks. They can either be end-to-end or peer-to-peer, and either one-step or two-step clocks. To be useful for NTP, I think it would need to be an end-to-end transparent clock and ideally it would be a one-step clock to avoid dealing with with the follow-up messages.

[cbhl]

It looks like the source for their FBOSS agent that runs on switches will enable PTP in two kinds of supported switches:

https://github.com/facebook/fboss/blob/master/fboss/agent/hw...

They have O(thousands) of switches (source: https://engineering.fb.com/2016/10/18/data-center-engineerin...) so IIRC they get the per-switch cost down by doing a custom design to reduce BOM. I believe the specs and designs are available under the Open Compute project here (search for "Facebook"): https://www.opencompute.org/wiki/Networking/SpecsAndDesigns

[jeffbee]

Even the retail price of 1 switch based on their "wedge" design is less than $10k for 32 100gbps ports. Merchant silicon like the Broadcom 56000-series switch ASICs support PTP.

[nonameiguess]

I suspect Facebook data centers were already using $20,000 network switches that supported PTP.

[mFixman]

I'm curious: what's the use case for microsecond-precise timekeeping?

[LeegleechN]

Globally distributed, consistent databases. See this Google paper: https://static.googleusercontent.com/media/research.google.c...

[hinkley]

There are consensus protocols that allow a grace period for clock skew, and the narrower you can constrain the clock skew, the more transactions per second you can retire, improving both scalability and latency.

[michaelt]

To monitor a control loop that ran in <1ms I needed something more precise than milliseconds.

I could have used an in-between time unit, like hundredths of a millisecond, but that would have created embarrassingly long variable names.

[VectorLock]

I was wondering this as well. FB tech articles never seem to start with a use-case, application or statement of problem.

[unsui]

The question I'm interested in a corrollary of the above question:

What's the use case for microsecond-precise timekeeping, that is of interest to (i.e., benefits) FB?

[lordnacho]

Certain trading operations require it. I think there's even a regulatory under Mifid2, bit I'm not in the know about the legals.

[xwolfi]

I work in trading, but mostly on the asian markets, so not sure about mifid. Usually the exchanges seem to give milli to second precision (which is very annoying: if Thailand says it's second-precision to timestamp their order executions but Taiwan says it's milli, our system has to use the most precise when ordering them and we get questions from our own multi market clients about why all these executions are seemingly simultaneous or why our system source marked the request at .480 while the exchange replied with 0.0 for the execution.

But, by no mean, does it mean we use special timing - but now that I say that, I realize I have no clue where our time signal comes from, clearly it wouldn't be the internet because our trading servers aren't exactly directly connected there, we must have an internal ntp sever o_O

[Bluecobra]

Not all exchanges are like this. The JPX exchange in Tokyo provides PTP over dedicated cross-connects, and is super accurate, within +/-25ns. It’s pretty neat seeing a PTP clock from Seiko.

[reportingsjr]

There are plenty of practical applications for it.

Imagine super high speed, high precision robotics, maybe a synthetic fiber layup machine or an exotic milling machine, that moves at 10m/s. If you have 1us of precision and accuracy then you can send movement commands that are precise down to 10um.

[eutectic]

But does that really require an absolute time reference?

[joshvm]

In these cases mostly not, they require low jitter references and a good counter.

An example that does - I work for IceCube, a big neutrino detector at the South Pole. The detector is essentially a time of flight system that detects emitted light as it propagates through the ice when a neutrino interaction happens. We need good time distribution in the system so we can marry up hits from the 5000 odd sensors in the ice (and successive hits may be nanoseconds apart), and we need good absolute precision because we collaborate with other observatories.

Also pretty much all radio telescopes that do long baseline interferometry need excellent absolute timing. The South Pole Telescope has (I think) the most accurate clock on station, a hydrogen maser-based system. We use GPS.

[eutectic]

Neat!

[hiyer]

I had the same question - there are obviously a lot of applications for this, but why would a company like FB need it?

[TravelPiglet]

Distributed databases I assume