Free lesson · System Design Fundamentals

The Clock Synchronization Problem

Medium10 min readsystem-designhlddistributed-systems

Lesson 20.1 - The Clock Synchronization Problem

The first lesson of Module 20.

You have two servers. Server A records an event at 2026-06-12T14:00:00.123Z. Server B records an event at 2026-06-12T14:00:00.119Z. Which event happened first? The obvious answer: Server B's event - it has the earlier timestamp. But that answer assumes both servers' clocks agree on what 14:00:00 means. What if Server A's clock is 10 milliseconds ahead of Server B's? Then Server B's event (which the real-world clock would say happened at 14:00:00.129Z after correcting for B's clock being slow) actually happened AFTER Server A's event. The timestamps lied. The ordering is wrong.

On a single machine, this problem doesn't exist - there's one clock. In a distributed system with dozens or thousands of machines, each with its own oscillator (a quartz crystal vibrating at approximately 32,768 Hz), the clocks drift apart. "Approximately" is the key word - quartz oscillators aren't perfect. They're affected by temperature, voltage, aging, and manufacturing variation. Typical drift: 10-100 parts per million (ppm), meaning a clock gaining or losing 10-100 microseconds per second - up to 8.6 seconds per day.

NTP (Network Time Protocol) synchronizes clocks across machines, reducing the drift. But NTP synchronization has inherent limitations: network latency is variable (the time it takes a clock correction packet to travel from the NTP server to the client varies by milliseconds), and NTP itself has accuracy limits (typically 1-10 ms over the internet, 0.1-1 ms within a datacenter). These milliseconds of uncertainty are the reason physical clocks can't be fully trusted for ordering events in distributed systems.

How Clocks Work in Computers

Every computer has two types of clocks:

Time-of-Day Clock (Wall Clock)

The familiar clock: Date.now() in JavaScript, System.currentTimeMillis() in Java, time.time() in Python. Returns the current date and time - synchronized to UTC via NTP.

Problems for distributed systems:

1. It can jump. NTP synchronization can adjust the clock forward or backward. If NTP determines the local clock is 200ms ahead, it steps the clock back by 200ms - meaning two consecutive Date.now() calls might return timestamps where the second is EARLIER than the first. This breaks any code that assumes timestamps are monotonically increasing.

2. Leap seconds. UTC occasionally inserts a "leap second" (the last one was in December 2016). During a leap second, the clock either repeats a second (23:59:60 → 23:59:59 → 00:00:00) or jumps. Some systems implement "leap smearing" (Google's approach) - spreading the extra second over a 24-hour period by slightly slowing the clock. During the smear period, Google's clocks and non-smearing clocks disagree by up to 500ms.

3. Clock skew between machines. Even with NTP, two machines' clocks can differ by 1-10ms (within a datacenter) or more (across the internet). This means comparing timestamps from different machines has a ±10ms uncertainty band - you can't reliably order events that happened within 10ms of each other.

Monotonic Clock

A counter that only moves forward: System.nanoTime() in Java, process.hrtime() in Node.js, clock_gettime(CLOCK_MONOTONIC) in Linux. It's NOT synchronized to UTC - it just counts elapsed time since an arbitrary start point (usually system boot). It NEVER jumps backward, even during NTP adjustments.

Used for: measuring durations (how long did this function take?), timeouts (has 3 seconds passed since we sent the request?), and any logic that needs "time elapsed" rather than "what time is it."

Limitation: monotonic clocks are local to one machine. Machine A's monotonic clock and Machine B's monotonic clock have no relationship - you can't compare them. They're useful for measuring durations on one machine, useless for ordering events across machines.

NTP: How Clock Synchronization Works

flowchart TD
    NTP["NTP Server\n(stratum 1 - connected to\nGPS/atomic clock source)\n\nKnows the 'true' time\nwith microsecond precision"]
    
    ClientA["Server A\nLocal clock: 14:00:00.000\n\n1. Sends NTP request\n   at local time T₁\n2. NTP server receives at T₂\n3. NTP server responds at T₃\n4. Server A receives at T₄\n\nOffset = ((T₂-T₁) + (T₃-T₄)) / 2\nRound-trip delay = (T₄-T₁) - (T₃-T₂)\n\nAdjust local clock by offset"]
    
    ClientB["Server B\nLocal clock: 14:00:00.008\n(8ms ahead of Server A)\n\nSame NTP synchronization\nprocess - corrects to\nwithin ±1ms of NTP server"]
    
    NTP --> ClientA
    NTP --> ClientB

    style NTP fill:#34d399,stroke:#10b981,color:#000
    style ClientA fill:#fbbf24,stroke:#f59e0b,color:#000
    style ClientB fill:#fbbf24,stroke:#f59e0b,color:#000

A diagram showing NTP clock synchronization. An NTP server (stratum 1 - directly connected to a GPS receiver or atomic clock) knows the true time with microsecond precision. Server A and Server B each synchronize with the NTP server independently. The NTP protocol works by exchanging timestamps: Server A records its local time when it sends the request (T1). The NTP server records when it receives the request (T2) and when it sends the response (T3). Server A records when it receives the response (T4). From these four timestamps, Server A calculates the clock offset (how far its clock is from the NTP server's) and the round-trip network delay. The offset formula accounts for the network delay - but it assumes the network delay is symmetric (the same in both directions). If the network delay is asymmetric (faster in one direction than the other - common with different network paths), the offset calculation is wrong by half the asymmetry. This asymmetry introduces unavoidable error - typically 0.1 to 10 milliseconds. After synchronization, Server A and Server B are each within approximately 1 millisecond of the NTP server's time - but they might be up to 2 milliseconds apart from each other (each ±1ms from truth). This 2ms uncertainty band means events on different servers that happen within 2ms of each other cannot be reliably ordered by their timestamps.

NTP accuracy in practice:

For events within the same datacenter: a 2ms uncertainty means you can reliably order events that are >2ms apart, but you CANNOT reliably order events within the 2ms window. In a system processing 50,000 writes per second, 100 writes happen within any 2ms window - any of those 100 writes could be misordered by timestamp.

When Clock Skew Causes Real Problems

Problem 1: Last-Writer-Wins Conflict Resolution

Many distributed databases use last-writer-wins (LWW) for conflict resolution: when two replicas have different values for the same key, the value with the latest timestamp wins. But if the clocks are skewed:

Server A (clock is 5ms fast):
  14:00:00.105 - writes key="name", value="Alice"

Server B (clock is accurate):  
  14:00:00.103 - writes key="name", value="Bob"

Real-world order: Bob's write happened 2ms AFTER Alice's write.
Timestamp order: Alice's write (105) > Bob's write (103).
LWW picks: Alice's write (higher timestamp).
Correct answer: Bob's write (happened later in real time).

Result: Bob's write is silently discarded. Data loss.

This is not a theoretical scenario - Cassandra uses LWW by default. With a 5ms clock skew, any writes within a 5ms window can be ordered incorrectly. At high write rates, this causes silent data loss that's nearly impossible to detect (the "lost" write was a valid, successful write - it just had a lower timestamp due to clock skew).

Problem 2: Distributed Lock Expiry

A distributed lock (used to prevent concurrent access to a shared resource) has a TTL - it expires after N seconds. The lock service's clock and the client's clock must agree on when the lock expires. If the client's clock is 2 seconds slow, the client thinks it has 2 seconds left on the lock when it's actually already expired - and another client has acquired the lock. Both clients proceed, believing they hold the exclusive lock. Mutual exclusion is violated.

Problem 3: Certificate and Token Expiry

TLS certificates and JWT tokens have notBefore and notAfter fields based on UTC time. If a server's clock is wrong (behind by 1 hour due to NTP failure), it might reject a valid certificate ("this certificate isn't valid yet") or accept an expired token ("this token is still valid"). In AWS, clock skew causes authentication failures with AWS APIs - AWS signatures include a timestamp, and AWS rejects requests with timestamps more than 5 minutes from AWS's clock.

Problem 4: Log Correlation

When debugging a distributed system, engineers correlate logs from multiple services: "Request entered Service A at 14:00:00.100, reached Service B at 14:00:00.095." Wait - Service B received the request BEFORE Service A sent it? That's impossible. The timestamps are wrong due to clock skew between Service A and Service B. Without reliable timestamps, distributed tracing (M25 covers OpenTelemetry) uses trace IDs and span IDs to correlate events - logical identifiers rather than physical timestamps.

Google's TrueTime: The Hardware Solution

Google recognized that NTP's millisecond-level uncertainty is too large for their globally distributed database (Spanner). Their solution: TrueTime - a clock API that returns a time INTERVAL, not a point.

NTP:     now() → 14:00:00.100   (a point - but how accurate?)
TrueTime: now() → [14:00:00.099, 14:00:00.101]   (an interval - guaranteed to contain the real time)

TrueTime uses GPS receivers and atomic clocks in every Google datacenter to reduce the uncertainty interval to ~7 microseconds (compared to NTP's ~1-10 milliseconds). Each datacenter has multiple independent time sources - if one GPS receiver fails, the atomic clock maintains accuracy for hours.

How Spanner uses TrueTime: when committing a transaction, Spanner assigns a timestamp within the TrueTime interval and then waits until the uncertainty interval has passed (the "commit wait"). After the wait, Spanner is certain that no other transaction on any machine in any datacenter will be assigned a timestamp that falls within or before this transaction's timestamp. This guarantees global ordering of transactions - even across continents.

The cost: every transaction pays a commit wait of ~7 microseconds (the TrueTime uncertainty bound). This is Google's trade-off: specialized hardware (GPS + atomic clocks, costing millions to deploy globally) in exchange for a clock that's accurate enough to order transactions globally without logical clocks.

CockroachDB (an open-source Spanner-inspired database) can't use TrueTime (no GPS/atomic clock infrastructure). It uses NTP for clock synchronization and configures a max-offset parameter (default: 500ms) - if any node's clock drifts more than 500ms from the cluster, it's ejected. CockroachDB's commit wait is proportional to max-offset - much larger than Spanner's 7μs, but workable with commodity NTP.

A Brandon Story: The Audit Log That Was Out of Order

A team I'd advised built an audit trail system - every user action was logged with a timestamp to a distributed audit log. Regulators required that the audit log show the correct sequence of events: "User viewed record at 14:00:00, modified it at 14:00:05, approved it at 14:00:10." The audit log was written by multiple application servers, each recording events with their local System.currentTimeMillis().

During an audit review, a regulator flagged an anomaly: the log showed a user APPROVING a document (14:00:08.200, Server C) BEFORE MODIFYING it (14:00:08.500, Server A). The modification happened on Server A (whose clock was 400ms fast), and the approval happened on Server C (whose clock was 100ms slow). Real-world order: modify first (14:00:08.100 real), approve second (14:00:08.300 real). Timestamp order: approve first (08.200), modify second (08.500). The audit log showed the impossible - approval before modification - because the clocks disagreed.

The fix: two changes.

1. Centralized timestamp authority. Instead of each server recording its local time, all audit events are sent to a centralized audit service that assigns timestamps from ONE clock. This eliminates cross-server clock skew - all events are timestamped by the same machine.

2. Logical ordering via sequence numbers. Each audit event includes a previous_event_id field - creating a causal chain. The approval event explicitly references the modification event as its predecessor. Even if timestamps are wrong, the causal chain shows the correct order: modify → approve. This is the logical clock principle (20.2).

Lessons:

• Physical timestamps from different machines can't be trusted for ordering events within the clock skew window (typically 1-10ms within a datacenter).
• Audit logs, financial records, and regulatory compliance systems need correct ordering - not just approximate ordering. Clock skew violates correctness guarantees.
• Centralized timestamp authority eliminates clock skew for ordering - at the cost of a single point of contention (all events funneled through one service).
• Causal ordering (event B references event A as its predecessor) is more reliable than timestamp ordering - it captures the "happened before" relationship explicitly, regardless of physical clocks.
• NTP synchronization (1-10ms accuracy) is sufficient for most applications. For ordering-critical systems, use logical clocks (20.2-20.4) or centralized timestamp assignment.

Six Takeaways from This Lesson

1. Physical clocks drift. Quartz oscillators gain or lose 10-100 microseconds per second. Without synchronization, clocks diverge by seconds per day.

2. NTP reduces drift but can't eliminate it. Accuracy: 0.1-1ms within a datacenter, 1-10ms across regions. The remaining uncertainty means events within the skew window can't be reliably ordered.

3. Wall clocks can jump (NTP adjustments, leap seconds). Monotonic clocks never jump but are local to one machine. Use wall clocks for "what time is it?" Use monotonic clocks for "how long has it been?"

4. Clock skew causes real bugs: last-writer-wins data loss, distributed lock violations, authentication failures, misordered audit logs.

5. Google's TrueTime reduces uncertainty to ~7 microseconds using GPS + atomic clocks - enabling globally ordered transactions in Spanner. No equivalent exists in commodity infrastructure.

6. The implication: physical clocks are unreliable for ordering events in distributed systems. Logical clocks (20.2) provide ordering without depending on physical time.

Try It

Exercise: Warmup (10 minutes). Check two machines' clocks (your laptop and a server, or two cloud instances). Run date -u +%s.%N on both simultaneously. How many milliseconds apart are they? Is NTP running on both? (ntpq -p or chronyc sources)

Exercise: Core (25 minutes). A distributed key-value store uses last-writer-wins with physical timestamps. Server A's clock is 5ms ahead. Two concurrent writes to the same key: Server A writes "Alice" at T=100ms (real time: 95ms), Server B writes "Bob" at T=98ms (real time: 98ms). Which write wins? Is that correct? Propose a solution that doesn't rely on physical clocks.

Exercise: Stretch (30 minutes). Read Google's Spanner paper (link below), focusing on Section 3 (TrueTime) and Section 4 (Concurrency Control). How does the commit-wait mechanism work? What happens if TrueTime's uncertainty bound increases (e.g., from 7μs to 100ms)?

What's Next

Lesson 20.2 introduces logical clocks - the idea of ordering events without depending on physical time at all.

Back to System Design Fundamentals