Physical Clocks: NTP, Clock Drift, Leap Seconds and Not Trusting the Wall Clock

Version note: The code in this post was built and run with C# 14 and .NET 10.

Guess First

Bookstore's catalog service runs on two servers and replicates book prices between them. Conflicting updates are resolved with the simplest rule there is: last-write-wins (LWW), meaning the write with the larger timestamp wins. At 10:00:00.000 an editor sets a book's price to 450 through server A. 100 milliseconds later the campaign team lowers the same book to 399 through server B. In real time, the second write is newer.

Server B's clock is 300 milliseconds behind A's. Once both replicas have received both writes, what is the book's price? Are the replicas consistent with each other? And is the result correct?

A Computer Has Two Clocks

"The clock" on a computer means two different things, and most time-related bugs in distributed systems come from mixing them up.

The time-of-day clock (wall clock) gives calendar time: 8 November 2026, 10:00:00 UTC (Coordinated Universal Time). It has to be comparable with other machines, so it is corrected from the outside with NTP (Network Time Protocol). During a correction it can jump forward or backward. In .NET, DateTime.UtcNow, DateTimeOffset.UtcNow and TimeProvider.GetUtcNow() read this clock.

The monotonic clock only moves forward. Its value has no absolute meaning (usually it is the time since the machine booted) and it cannot be compared across machines; its only job is to measure the time between two instants on the same machine. NTP may slightly adjust its rate but never steps it back. In .NET, Stopwatch, Stopwatch.GetTimestamp(), Environment.TickCount64 and TimeProvider.GetTimestamp() read this clock; underneath are operating system counters such as QueryPerformanceCounter on Windows and CLOCK_MONOTONIC on Linux.

The hardware behind both clocks is usually a quartz crystal, and quartz drifts (clock drift). The crystal's frequency varies a little with manufacturing and temperature; typical drift is in the order of tens of parts per million (ppm). 50 ppm looks small but adds up to about 4.3 seconds a day. A server that is never synchronized can be half a minute off after a week.

How, and How Well, NTP Corrects the Clock

NTP is the protocol David Mills designed in the 1980s, today defined by RFC (Request for Comments) 5905. Servers are arranged in layers (strata): stratum 0 is an atomic clock or a GPS (Global Positioning System) receiver, stratum 1 is a server attached directly to it, stratum 2 gets its time from stratum 1, and so on. A client sends a request to a server, estimates the network delay from the outbound and return times, and computes how far its own clock is off (the offset). The estimate assumes the outbound and return delays are equal; an asymmetric path (for example a saturated upload link) breaks that assumption and the error goes straight into the offset.

These are the real answers I got when I asked two different NTP servers from the machine I am writing this on (the sntp tool on macOS; the first number is the offset in seconds, the second the error bound):

$ sntp time.apple.com
-0.001019 +/- 0.043825 time.apple.com 17.253.14.45
$ sntp pool.ntp.org
-0.012952 +/- 0.037462 pool.ntp.org 162.159.200.1

Two servers asked at the same time disagree about my machine's clock by 12 milliseconds, and both give an error bound of about ±40 milliseconds. That is the practical accuracy of NTP over the internet; within one data center, with a nearby server, it can go below a millisecond. Bad days happen too: the NTP server is unreachable, a firewall drops the packets, a misconfigured server pushes clocks years ahead.

If the offset is small, the NTP daemon does not jump the clock; it closes the gap slowly by changing its rate slightly (slewing). If the offset is large, it steps the clock forward or backward in one go; the classic ntpd's default threshold is 128 milliseconds. When a virtual machine is paused for a long time and resumed, when a server boots for the first time, or when someone fixes the clock by hand, the wall clock suddenly jumps. That is when it starts to matter what your duration-measuring code does.

Measuring Durations with the Wall Clock in .NET

Instead of waiting for NTP to actually step the clock back, let us simulate it with TimeProvider. This abstract class, introduced in .NET 8, exposes the wall clock (GetUtcNow) and the monotonic timestamp (GetTimestamp) separately. Our own provider can add an offset to the wall clock and leaves the timestamp alone:

namespace Bookstore.Clocks;

// Wall clock that NTP can step; the monotonic timestamp is untouched
public sealed class SteppingTimeProvider : TimeProvider
{
    private TimeSpan _offset;

    public void Step(TimeSpan delta) => _offset += delta;

    public override DateTimeOffset GetUtcNow() => System.GetUtcNow() + _offset;

    public override long GetTimestamp() => System.GetTimestamp();

    public override long TimestampFrequency => System.TimestampFrequency;
}

We measure a task that takes about 200 milliseconds in two ways; halfway through, NTP sets the clock back by 300 milliseconds:

using System.Diagnostics;
using System.Globalization;
using Bookstore.Clocks;

CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;

Console.WriteLine($"Stopwatch.IsHighResolution = {Stopwatch.IsHighResolution}");
Console.WriteLine($"Stopwatch.Frequency        = {Stopwatch.Frequency:N0} ticks/s");
Console.WriteLine($"TimeProvider.System.TimestampFrequency = {TimeProvider.System.TimestampFrequency:N0}");

// 1) Measuring a duration while NTP steps the clock back by 300 ms
SteppingTimeProvider clock = new();
DateTimeOffset wallStart = clock.GetUtcNow();
long monoStart = clock.GetTimestamp();

await Task.Delay(200);           // the real work takes about 200 ms
clock.Step(TimeSpan.FromMilliseconds(-300)); // NTP correction happens in the middle

TimeSpan byWallClock = clock.GetUtcNow() - wallStart;
TimeSpan byMonotonic = clock.GetElapsedTime(monoStart);
Console.WriteLine($"wall clock elapsed : {byWallClock.TotalMilliseconds:F0} ms");
Console.WriteLine($"monotonic elapsed  : {byMonotonic.TotalMilliseconds:F0} ms");

Stopwatch.IsHighResolution = True
Stopwatch.Frequency        = 1,000,000,000 ticks/s
TimeProvider.System.TimestampFrequency = 1,000,000,000
wall clock elapsed : -97 ms
monotonic elapsed  : 203 ms

The first three lines show that this machine's monotonic counter has nanosecond resolution. Then the real lesson: according to the wall clock, the task took minus 97 milliseconds. If that number lands on a metrics dashboard, the chart breaks; if it goes into a timeout calculation, the timeout never fires; if it goes into a rate limiter, the limit resets. In the other direction, if the clock jumps forward, a 200-millisecond task appears to take 5 seconds and a healthy request hits a timeout. The monotonic measurement tells the truth: 203 milliseconds.

This bug has taken down real systems. During the leap second on 1 January 2017, a Go program in Cloudflare's DNS (Domain Name System) infrastructure computed a negative duration from the wall clock, passed a negative value to a random number function and crashed with a panic; the company's incident report pins the problem precisely on the assumption that "time cannot go backwards". The rule is simple: durations, timeouts, rate limits, leases and cache ages are measured with the monotonic clock. The wall clock is only for calendar time that is shown to people or exchanged with other machines.

Last-Write-Wins and the Lost Update

Back to the opening question. An LWW register is a few lines:

namespace Bookstore.Clocks;

public record Versioned(string Value, DateTimeOffset Timestamp, string Node);

// Last-write-wins: the higher timestamp survives, the other write disappears
public sealed class LwwRegister
{
    public Versioned? Current { get; private set; }

    public void Apply(Versioned write)
    {
        if (Current is null || write.Timestamp > Current.Timestamp)
        {
            Current = write;
        }
    }
}

A's clock is right, B's is 300 milliseconds behind. In real time the second write comes 100 milliseconds later. The two replicas receive the two writes in different orders:

// 2) Last-write-wins between two servers, server B is 300 ms behind
DateTimeOffset trueTime = new(2026, 11, 8, 10, 0, 0, TimeSpan.Zero);
TimeSpan skewA = TimeSpan.Zero;
TimeSpan skewB = TimeSpan.FromMilliseconds(-300);

Versioned first = new("price=450", trueTime + skewA, "A");
Versioned second = new("price=399", trueTime.AddMilliseconds(100) + skewB, "B"); // 100 ms later in reality

LwwRegister replica1 = new();
LwwRegister replica2 = new();
replica1.Apply(first);
replica1.Apply(second);
replica2.Apply(second);
replica2.Apply(first);

Console.WriteLine($"write 1 on A: {first.Value} ts={first.Timestamp:HH:mm:ss.fff}");
Console.WriteLine($"write 2 on B: {second.Value} ts={second.Timestamp:HH:mm:ss.fff} (really 100 ms later)");
Console.WriteLine($"replica 1 keeps: {replica1.Current?.Value} from {replica1.Current?.Node}");
Console.WriteLine($"replica 2 keeps: {replica2.Current?.Value} from {replica2.Current?.Node}");

write 1 on A: price=450 ts=10:00:00.000
write 2 on B: price=399 ts=09:59:59.800 (really 100 ms later)
replica 1 keeps: price=450 from A
replica 2 keeps: price=450 from A

Both replicas settled on 450. The system is consistent (both copies show the same value), but wrong: the campaign price vanished without an error or a single log line. B's write got the timestamp 09:59:59.800, because that is what B's clock said at the time. LWW does not pick the "last write" but the "write with the largest timestamp", and those are the same thing only while clocks are perfectly synchronized.

This is not just a textbook scenario. In databases that use LWW, such as Cassandra, each cell's timestamp comes from the client or the coordinator node that performed the write; an application server whose clock lags cannot overwrite data others have just written, and one whose clock runs ahead makes every later write look "old". Kleppmann's DDIA (Designing Data-Intensive Applications) uses this case as an example of how ordering in replication can come out wrong even with NTP-synchronized clocks. The solutions are the subject of the next posts: logical clocks (Lamport, vector clocks) to order events, version vectors to detect and keep conflicts, and CRDTs (Conflict-free Replicated Data Types) to make timestamps unnecessary through mergeable data types.

Leap Seconds and Smearing

The Earth's rotation is irregular and slowing down; to keep atomic-clock-based UTC close to astronomical time, a second has been added to UTC 27 times since 1972. The last of these leap seconds was inserted on 31 December 2016, as 23:59:60. A minute that lasts 61 seconds threatens every piece of code that assumes seconds run from 0 to 59; in .NET, DateTime cannot represent 23:59:60. During the 2012 leap second, a bug in the Linux kernel drove the CPU (Central Processing Unit) usage of processes such as Java and MySQL to one hundred percent and caused outages at many large sites.

Large cloud providers solve the problem with a leap smear: instead of inserting the extra second at one instant, they spread it over 24 hours, as Google does, running clocks slightly slow during that day. The clock never jumps, but during the smear it differs from true UTC by up to half a second; mix smearing and non-smearing servers and that difference turns straight into clock skew. In the long run the problem goes away: in 2022 the General Conference on Weights and Measures (CGPM) decided to stop adding leap seconds by 2035 at the latest.

Testable Time with TimeProvider

The real payoff of TimeProvider is in tests. To test a time-based rule (a cart reservation expires after 15 minutes), instead of waiting 15 minutes or wrapping DateTime.UtcNow in a static wrapper, you take the provider through the constructor:

namespace Bookstore.Clocks;

// A cart reservation that expires after 15 minutes, measured with a monotonic timestamp
public sealed class ReservationHold(TimeProvider time)
{
    private readonly long _startedAt = time.GetTimestamp();

    public bool IsExpired => time.GetElapsedTime(_startedAt) >= TimeSpan.FromMinutes(15);
}

In a test, FakeTimeProvider from the Microsoft.Extensions.TimeProvider.Testing package advances time by hand:

// 3) Testing time-dependent code without sleeping
Microsoft.Extensions.Time.Testing.FakeTimeProvider fake = new();
ReservationHold hold = new(fake);
fake.Advance(TimeSpan.FromMinutes(14));
Console.WriteLine($"after 14 min expired? {hold.IsExpired}");
fake.Advance(TimeSpan.FromMinutes(1));
Console.WriteLine($"after 15 min expired? {hold.IsExpired}");

after 14 min expired? False
after 15 min expired? True

In production you inject TimeProvider.System. Because the same provider can be passed to overloads of Task.Delay, CancellationTokenSource and PeriodicTimer, code that uses timers can be tested in virtual time as well. The C# Async series covers this API (Application Programming Interface) in detail in "C# Async and .NET 8: TimeProvider, ConfigureAwaitOptions, Parallel.ForAsync and CancelAsync".

Common Pitfalls

Measuring durations with DateTime.Now. It is the wall clock and local time; on top of NTP jumps it leaps an hour at daylight saving transitions. For durations use Stopwatch.GetTimestamp() with Stopwatch.GetElapsedTime, or TimeProvider.GetElapsedTime.

Ordering events by timestamp. Comparing two timestamps produced on two different machines is meaningless for intervals smaller than the difference between their clocks. Remember this when merging logs or sorting events by "which came first"; if you need an order, it must come from a single sequencer (a database sequence, a broker partition offset) or a logical clock.

Not monitoring clock synchronization. The NTP daemon can quietly stop working. Collect the clock offset as a metric and alert when it crosses a threshold; some databases, such as CockroachDB, shut a node down themselves if its offset grows too large.

Moving monotonic values between machines. The value of Stopwatch.GetTimestamp() means nothing on another machine, and it may start from zero again after a restart. Any time value that goes to a database or another service is wall clock time and carries the uncertainty above.

Computing leases and expiry against another machine's clock. "The lock is mine until 10:00:30" is relative to the clock of the server that granted it; if your clock is 2 seconds ahead you give the lock up early, if it is behind you keep writing after it has expired. That is exactly the subject of the tenth post.

A Small Experiment

★☆☆ Change skewB from -300 milliseconds to -50 milliseconds. Does the campaign price survive this time? Below what clock difference does LWW give the right answer here?

★★☆ Use SteppingTimeProvider to jump the clock 2 seconds forward, and write a small rate limiter that enforces "reject if more than 5 requests arrived in the last second" with the wall clock. What happens after the jump? Rewrite the same rule with GetTimestamp.

★★★ Add the case of two writes getting the same timestamp to the LWW register (for example both servers produce 10:00:00.000). How do you break the tie so the replicas converge on the same value? Does that rule guarantee the "right" value is chosen?

In this post we saw why physical clocks are not good enough for ordering events. In the next post we build an order without looking at a clock at all: Lamport's happens-before relation, Lamport clocks, vector clocks that can detect concurrent events, and small C# implementations tested with xUnit.