Announcing: Statime, a Rust PTP implementation

PTP is a method of accurate time synchronization between computer systems. It is used when extreme time precision is required, for instance, in physics experiments, car control systems, finance, broadcasting, security protocols, smart grids, and in 5G.

The project's current status is that we have a working proof of concept; see the project's github repository. Today, only Linux is supported, but the project is set up to support other platforms in the future easily (e.g., embedded platforms).

In the rest of this blog post, we'll describe our objectives and the development process. We'll also go into some of the technical details.

What is PTP?

PTP is an acronym that stands for Precision Time Protocol. It synchronizes the time between two or more computer systems very precisely, sometimes even into the nanoseconds range. More common time protocols like NTP (Network Time Protocol) can't reach this precision.

If synchronizing is not actively done, then the clocks of the computers will drift apart due to minor differences in hardware and even temperature.

There are two different ways in which two clocks can be different from each other, namely their time offset and the frequency (how fast they think a second is).

PTP can measure these differences and correct the clocks.

Usage

PTP is designed to be deployed in both small and extensive networks using PTP-enabled devices. There are three types of devices:

• Grandmasters
  The source of the time. A grandmaster gets its time from a GPS signal or an atomic clock.
• Transparent clocks/switches
  A normal network switch that timestamps incoming messages and timestamps outgoing messages. Any measured delay in the message is corrected by writing to the message's correction field to compensate for this time
• Boundary clocks/switches
  These switches have their own internal clock that they synchronize with a master. On all other ports, it acts as a master itself.

Overview of a PTP network

How it works

By default, PTP uses multicast messages: every device on the same network can hear all of the messages.

All devices that can act as a PTP master will send out an announce message every once in a while, identifying them to all other devices on the network.

All PTP devices have some algorithms to determine if they need to act as a master or as a slave.

A device in the master state will send out Sync messages on a fixed frequency. The time this message is sent denotes T1. A receiving device will note down the time of the reception as T2. The slave now knows the time of the master and its own time, but not the delay between them.

The slave then sends a delay request to the server to determine the network delay. The sending timestamp is now T3. The server receives this message and the time is recorded as T4 and then sent to the slave using a delay response message.

Now the slave knows four different timestamps.

From these timestamps, we need to derive two things:

• What is the time offset to the master?
• What is the delay to the master?

Luckily we have a lot of information now:

MasterOffset + NetworkDelay = T2 - T1
-MasterOffset + NetworkDelay = T4 - T3

So to get the master offset, we can do the following:

NetworkDelay = ((T2 - T1) + (T4 - T3)) / 2
MasterOffset = (T2 - T1) - NetworkDelay

IEEE

PTP is an IEEE protocol and its last update was in 2019. Wikipedia says it best:

The original version of PTP, IEEE 1588-2002, was published in 2002. IEEE 1588-2008, also known as PTP Version 2 is not backward compatible with the original 2002 version. IEEE 1588-2019 was published in November 2019 and includes backward-compatible improvements to the 2008 publication.

Every standard is published as a big pdf with all the implementation details.

Our development

Implementing the full PTP specification is no small endeavor. Our first step was defining the scope for a feasible first phase. We decided to focus the initial prototype on an IP/UDP implementation with support for synchronizing time as a slave device and implemented on a (desktop-class) Linux OS.

Even though the prototype will be limited, it should be general enough to be used in all kinds of scenarios like Windows and embedded platforms.

Our motivation for the project was twofold:

• To build up knowledge of the network timing domain, as we were also involved in another timing-related project.
• To see if we can contribute to infrastructure safety by writing a modern and memory-safe implementation of a widely-used piece of software.

NLnet grant

Before the development started, we needed to ensure that we got funding. As the project would be open source and we'd make no money from people using the implementation, we searched for relevant grants.

The non-profit NLnet Foundation was formally started in 1989 and had significant early achievements in the Dutch internet.

• The first internet backbone in Holland.
• The first local dial-in and ISDN infrastructure with full country coverage.
• The definition and implementation of a low-cost connectivity structure.

Currently, the foundation focuses on creating a safer and more secure internet. They regularly have funds available, for instance, in the Internet Hardening Fund and the NGI Assure themes.

Existing open-source implementations of PTP are all written in C, which causes them to have potential security vulnerabilities. A memory-safe Rust implementation fits with the goals of NLnet. In general, binding to an unsafe implementation is not a logical choice when future developments focus more and more on safety and security.

Luckily NLnet agreed with us, and after our project was approved, we were ready to start.

Process

The PTP protocol consists of different components that are relatively standalone. That suited our plan well to work with multiple people from the start. Our team consisted of three developers, and at the beginning, we each took up one task:

• Ruben started working on kernel timestamping: the operating system needs to record the time at which a message was sent or received
• David made sure we had a Raspberry Pi with a PPS (pulse per second) output and put together a test setup to verify our implementation
• I started with creating the common data structures and messages of the protocol

Test setup with a Basys-3 FPGA development board setup to measure pps pulses, a Raspberry Pi 4 compute module with an Intel i210 NIC, and the Endrun Ninja Grandmaster clock synchronized via GPS.

As the weeks went by, we finished more components, and our implementation kept evolving. Every time we completed a new component, it would be integrated into our runtime until the protocol was implemented.

Problems

Of course, not everything was smooth sailing.

Most of the weird things we had to deal with had to do with documentation. Firstly, the IEEE document has sentences that can be interpreted in two different ways, and secondly, (good) documentation for some Linux APIs was lacking.

For example, let's take a look at the Linux API for adjusting the system clock: https://manpages.debian.org/testing/manpages-dev/ntp_adjtime.3.en.html

At first glance, everything seems to be in order here. Every item is documented. But once you dig in, you'll see some of the problems. Let's look at one of the status flags.

STA_PPSWANDER (read-only)
    PPS signal wander exceeded.

I know what each word means, but I don't know what the sentence means. So, signal wander... To which reference? What has exceeded what? And by how much?

This API was also the last major thing that we hadn't been able to solve. If we used it to adjust the clock of our PTP-enabled network card, it simply didn't change. The API allows you to do multiple things at once, and we used that to make frequency and time offset changes in one call. However, something in the system would ignore both if we did that. The fix was to do this change in two calls instead of one.

Results

In the end, we have a working slave-only E2E PTP library and an executable for Linux.

Accuracy

The objective of this first phase wasn't to produce a competitive implementation in terms of precision, but of course, we wanted to see what we could accomplish with our proof of concept. Beforehand we had set our goal to 5 microseconds.

To measure the accuracy of our implementation, we went to the Dutch national metrological institute VSL. More specifically, we visited the laboratory that houses the Dutch national standard of time.

Image of the VSL lab with their atomic clocks and our tiny EndRun PTP master clock

They have some costly measurement devices there that we were able to use.

At that time of the visit, we did not have hardware timestamping (a stretch goal) implemented, so we did our tests with kernel timestamping.

In our setup, we had some port asymmetry. Port asymmetry is where the path to and from the PTP master has different timing and is something that can be tuned out. Our constant offset (the asymmetry) from the real time was 23.5 microseconds.

Most important, of course, was the measurement of the deviation from that constant time. 90% of all observations fall within 11.2 microseconds of the constant time.

Given that we obtained this result with kernel timestamping, we were pretty happy with it.

Improved accuracy with hardware timestamping

Since our visit to the lab, we added hardware timestamping to our implementation, enabling us to measure the time at which messages are sent and received in hardware. Hardware timestamping is much more accurate than kernel timestamping. We repeated the measurement with our setup, and the results were very satisfying. A full report describing our measurements and its results is on our github. A short summary of the main results can be seen below.

Summary of results:

The accuracy is well below 2 microseconds, with which we reached our accuracy goal (5 microseconds).

To compete with Linux PTP there is obviously still some work to do. As we have spent little time on optimizing accuracy, we expect to be able to make big improvements when we do.

The future

Currently, the implementation is minimal. We have a slave-only E2E implementation that runs on Linux. It can be ported to, e.g., Windows and other hosted environments.

The specification includes many more features like Master mode, P2P support, and different profiles for, for instance, telecommunications and electric power distribution.

The upside is we have done a big chunk of the groundwork that allows us to add these features faster.

In the next phase our goal is to:

• Extend the project with master mode to cover all major PTP features, and
• add no_std support to allow running on embedded microcontrollers.

We will likely pair that with an implementation for the PTP-support in the STM32 PTP peripheral. Apart from expanding the feature set, we will work on improving the code quality and accuracy in every iteration we do.

Parallel to these efforts, we are in talks to use our software in field tests later this year.

Keep an eye on future developments on the project's GitHub page. We will be looking for sponsors and partners to continue our implementation. Feel free to reach out to us if you're interested.

Statime is part of The Pendulum project. Pendulum is building modern, open-source implementations of the Network Time Protocol and the Precision Time Protocol.