[jonathanyc]

Does anyone know the differences between Meta's application of Precision Time Protocol and Google TrueTime? I was hoping to find some discussion in the article but found none.

The 2022 article on the Precision Time Protocol says (https://engineering.fb.com/2022/11/21/production-engineering...):

- Adding precise and reliable timestamps on a back end and replicas allows us to simply wait until the replica catches up with the read timestamp...

- As you may see, the API doesn’t return the current time (aka time.Now()). Instead, it returns a window of time which contains the actual time with a very high degree of probability...

Which sounds similar to TrueTime (https://static.googleusercontent.com/media/research.google.c...):

- A read-only transaction executes in two phases: assign a timestamp sread [8], and then execute the transaction’s reads as snapshot reads at sread. The snapshot reads can execute at any replicas that are sufficiently up-to-date...

- TT.now() returns TTinterval: [earliest, latest]

I tried Googling "Precision Time Protocol TrueTime" but the only reference I could find is a HN comment by someone else from 2022 :) https://news.ycombinator.com/item?id=33696752

[aseipp]

PTP is part of the backbone of something like TrueTime. Meta uses their PTP infrastructure for a lot of the same basic fundamentals, including consistent read replicas, like Spanner does. PTP is a protocol for synchronizing the wall clock time of a set of computers under very tight error bounds, so that all of the servers have a very consistent and tight "view" of what time it is.

Now, independently of replica strategies, it's important to understand TrueTime is an API, to be clear, as you noted. It lets you represent some continuous interval of time based on the system clock error. You can then use this API to do things like ask "Did timestamp A occur before B?" And you can get an API equivalent to TrueTime on your own random Linux machine, using the Clock-Bound tools from AWS, combined with the chrony NTP daemon: https://github.com/aws/clock-bound

The API and all that is pretty basic, actually. Rather, the secret behind TrueTime and the like is just a huge amount of reliability engineering to ensure that the upper bound target (7ms IIRC from the Spanner paper) is actually maintained reliably and accurately, at global scale. That reliability means engineers can build on it with specific guarantees. You can slap chrony, ClockBound-D, and a PTP card into your rack and program away. But it's a matter of engineering guarantees more than like, theoretical computer science. Theoretically speaking, TrueTime can only help you definitively establish that some event A has actually "happened before" B in a distributed system. That's extremely powerful but it needs muscle backing it up to be true and useful in practice.

AWS has publicly advertised that EC2 has access to their 'AWS Time Sync' service, which is a globally consistent clock synchronization service designed to provide the backbone needed for services like TrueTime, and is freely available. Assuming you are willing to trust the EC2 network and AWS engineers, you can slap chrony and ClockBound-D on your AWS instances and get a TrueTime-like API with very tight global error tolerance, which would allow you to do consistent read replicas like Spanner, among other tricks

[j16sdiz]

TrueTime is designed for monotonicity. It give bounds for concurrency control.

PTP do wallclock time.

[touisteur]

Just one slight trapdoor I triggered in my travails. I don't know whether PTP does PLL slaving but while NTP is also for wall time, it has an effect on clock_gettime(CLOCK_MONOTONIC) so if you need a local clock without external (network, I mean) influence, CLOCK_MONOTONIC_RAW is there for you. And luckily, on recent kernel it has been vDSO'd.

[mjb]

Doing consistent reads this way is a fairly old technique, at least back to the 80s (I can dig up some papers if folks are interested).

Spanner, rather famously, uses this range approach, but a good number of other systems are based on similar approaches. The important thing for reads is getting an upper error bound from the clock, having storage that can perform reads at that time (eg using MVCC), and having a way for storage to know when it's seen all writes before a timestamp.

This can be implemented using PTP, and that's the approach we use at AWS inside some database services.

[jonathanyc]

Thank you! I’d be very interested in any papers you have the time to dig up. I feel like even one paper would give me a good pointer into the literature from which to start exploring.