AlgoInsight

System Design

Heartbeats: How Distributed Systems Stay Alive

A node goes silent. Is it dead, or just slow? Your whole system's stability rests on how it answers that one deceptively hard question.

By Admin2026-06-189 min readheartbeatsdistributed-systemsfailure-detectionreliabilityhigh-availabilitysystem-design

Here's a question that sounds simple and absolutely is not: in a system of a thousand machines, when one of them dies, how do the others find out?

There's no alarm that rings. A crashed server doesn't send a farewell message - it just goes silent. And silence is ambiguous. A node that's stopped responding might be dead. Or it might be perfectly alive but stuck behind a slow network link, or paused mid-garbage-collection, or just overloaded for a second. The entire stability of a distributed system rests on answering one question correctly and quickly: is that node dead, or just slow? Get it wrong in either direction and you have a problem. This is failure detection, the necessary first step before any of the failure handling I've written about elsewhere can kick in.

The heartbeat

The standard mechanism is beautifully simple. Each node periodically sends a small "I'm alive" message - a heartbeat - to whoever is monitoring it (a central monitor, or its peers). As long as the heartbeats keep arriving on schedule, everyone assumes the node is healthy. When the heartbeats stop, the monitor concludes the node has failed and reacts - reroutes its traffic, promotes a replica, triggers a leader election, rebalances its data.

sequenceDiagram
    participant N as Node B
    participant M as Monitor
    N->>M: heartbeat (t=0)
    N->>M: heartbeat (t=1)
    N->>M: heartbeat (t=2)
    Note over N: Node B crashes 💥
    Note over M: t=3 - no beat
    Note over M: t=4 - no beat (timeout exceeded)
    M->>M: mark Node B DEAD → trigger failover

This sequence diagram shows heartbeat-based failure detection. Node B sends heartbeats to a monitor at times zero, one, and two. Then Node B crashes. The monitor sees no heartbeat at time three, and again none at time four, exceeding its timeout. The monitor then marks Node B as dead and triggers failover.

That's the whole concept. The difficulty - and where the real engineering lives - is in one number: how long do you wait before declaring a node dead?

The dead-vs-slow dilemma

That timeout is a genuine trade-off with no perfect setting, and you can feel both failure modes by sliding it:

HEARTBEAT_INTERVAL = 1.0    # node beats once per second

def verdict(last_seen, now, timeout):
    return "DEAD" if (now - last_seen) > timeout else "ALIVE"

# A node's last heartbeat arrived at t=10.0. It's now t=12.5 - about 2 beats missed.
last_seen, now = 10.0, 12.5
for timeout in [1.5, 3.0, 5.0]:
    print(f"timeout={timeout}s -> node marked {verdict(last_seen, now, timeout)}")

Run it. With a short timeout (1.5s) the node is declared DEAD after missing barely one beat - fast detection, but dangerous: if that node was merely slow for a moment, you just marked a healthy node dead and triggered an unnecessary failover. With a long timeout (5s) the same node is still considered ALIVE - safe against false alarms, but if it really is dead, you'll keep routing traffic to a corpse for five full seconds.

Important: This is the core tension of failure detection:

  • Timeout too short → false positives. You declare healthy-but-slow nodes dead, causing needless failovers, data reshuffling, and system churn - sometimes a cascade where the churn itself slows nodes down and gets them falsely marked dead.
  • Timeout too long → slow detection. Dead nodes keep receiving requests that silently fail, and recovery is sluggish.

There is no universally correct value. It's tuned to your network's real-world latency and how costly a wrong guess is.

I learned to respect this the hard way: an aggressive timeout that looked great in testing caused a storm of false failovers in production the first time the network had a bad few seconds. Tuning failure detectors is unglamorous and it matters enormously.

Smarter detection

Because a fixed timeout is so blunt, real systems get cleverer:

Interview: If a design question involves a cluster of nodes - a distributed database, a leader-follower setup, a coordinator managing workers - mentioning heartbeats unprompted is a strong signal. Even better: "I'd detect node failure with heartbeats, and I'd tune the timeout carefully - too aggressive and a slow node gets falsely marked dead, triggering unnecessary failover." That one caveat shows you understand detection isn't free.

Detection is step one, not the whole story

Heartbeats answer "is it down?" Everything that happens next - rerouting traffic, electing a new leader, promoting a replica, rebalancing data - is failure handling, which is its own toolkit (timeouts, circuit breakers, graceful degradation, and the rest). But none of it can start until detection fires, and detection is only as good as that timeout you chose. Get the heartbeat wrong and you either react to failures that aren't real or miss the ones that are.

It's a small mechanism doing a load-bearing job - which is exactly why it shows up in nearly every distributed system worth studying. We connect detection and handling into one coherent picture, with the tuning trade-offs spelled out, across System Design Fundamentals.

Go deeper

More articles