Target practice: ntpd-rs on FreeBSD and other platforms

Most of our code is written in a target-agnostic way. For most code, any unix-like target will do. But because our lowest layer interfaces directly with libc, some subtle changes may be needed to support a new target. For instance, some types have slightly different fields, or more modern Linux APIs don't have an equivalent.

Sockets

We use libc and usafe code to configure UDP sockets, so that they provide timestamps when sending or receiving messages. Linux provides several APIs for timestamping and we use the most modern and accurate method. The other unix-like systems don't have a direct equivalent.

Memory Layout

Musl makes some minor changes to memory layout of some types. It adds private padding fields, so it is impossible to initialize a value of those types using only safe code. We have to use a bit of unsafe code instead:

fn empty_msghdr() -> libc::msghdr {
    unsafe { MaybeUninit::<libc::msghdr>::zeroed().assume_init() }
}

Initializing with zeros should be fine for all libc types, but it certainly is for the particular ones that we use. In fact we must zero the memory here; just using MaybeUninit::uninit would trigger undefined behavior (UB), because using uninitialized memory as an integer/pointer is undefined behavior. This might be slightly unintuitive because the bits form a valid integer value. Nonetheless, the decision was made to declare this UB.

Timestamping Method

MacOS does not provide a method for nanosecond timestamps, it only supports the older posix method of microsecond timestamps, which are sufficiently accurate for NTP clients. MacOS systems are not designed to be (NTP) servers.

FreeBSD has its own nanosecond precision API, which is sligtly different from the one on Linux. An extra flag can be set, and from then on the timestamps will contain nanoseconds instead of microseconds.

Clock steering

FreeBSD and macOS use an older API to update the clock. Fortunately, Linux still supports this API, so the implementation also works on Linux, and it was easy to develop on Linux.

The major downside of these APIs is that many operations require more syscalls. For instance stepping the clock on Linux can take a delta, but on macOS and FreeBSD requires the full timestamp. That means that before setting the clock, you need to first read the clock to get the current time, then add the delta, then set the clock again.

Time elapses between reading and subsequently setting the clock, which reduces precision. The modern Linux APIs use just one atomic update to modify the current time and are therefore more precise.

Pool deployment

The code changes to support these 3 new platforms were relatively straightforward. But (sadly) the fact that ntpd-rs compiles for these targets does not guarantee that it actually works. We need to validate the behavior by actually running the code on the target system.

We now run our tests for all targets, to make sure all primitives work as expected. Then we also perform manual tests where we set the time forward or backward by some amount and observe that the clock is re-synchronized to the correct time.

Finally, to get some production experience and verify that our implementation performs well for real traffic, we added a FreeBSD server to the NTP pool.

Building for FreeBSD

I develop on Linux, and up to this point had not actually used FreeBSD. All of our CI machines are also Linux machines. But to deploy to the pool, we'd need a FreeBSD build of ntpd-rs. We also want this build to be reproducable.

Rust supports many targets, but the cross-compilation story is still rough. A major pain point is that there is no convenient way to acquire a linker and system libraries for the target platform.

Zig does much better: it bundles many linkers and system library headers into its binary, and it is my prefered way to do cross-compilation. cargo-zigbuild is a wrapper around cargo that uses zig as the linker. We use cargo-zigbuild in our CI to run clippy for some targets like 32-bit Linux for the raspberry pi.

However, FreeBSD is not currently supported by cargo-zigbuild, and I ran into missing libraries trying to make it work. I'm sure it's possible, but I have no patience for this.

So instead we use an inelegant Vagrant setup to spin up a FreeBSD VM, install Rust, and build our source code. I'm not proud of it, it still took a day to set up, but it works and the build is now reproducable.

Configuration

Next I set up a FreeBSD machine in Google Cloud. Then it is just a matter of putting the right files in the right places. We use an Ansible Playbook to move all of the files over.

The final missing ingredient is how to run ntpd-rs as a daemon. On Linux we use systemd to take care of the details, but FreeBSD does not have a direct equivalent. We eventually got a working /etc/rc.d/ntp-deamon configuration that has all of the behavior we want (like running in the background and rotating log files).

With our daemon set up, it was time to add our new server to the NTP pool and slowly watch its accuracy score climb.

ntpd-rs on FreeBSD slowly reaching a score of 20 in the pool

It looks like the output is a bit noisier than for our Linux servers, but it is more than adequate for the NTP pool.

Conclusion

It is very exciting that we support so many targets now. The FreeBSD compilation process is rough, but overall the process went smoothly. We are very happy to see such substantial community contributions!

Try ntpd-rs today. For Linux we provide installers. For FreeBSD this PR contains useful information.

We would greatly appreciate feedback from users, particlarly those who are running ntpd-rs on a sizable network of computers. Opening an issue to share feedback or suggestions is an option, but you may also contact Erik directly with your issues or proposed contributions.