AlgoInsight

Worked system design solution

Design a Health Check / Heartbeat Service

A service that monitors thousands of distributed nodes by detecting "is this node alive?" within seconds of failure. We design active vs passive heartbeats, hysteresis to avoid flapping, and the gossip pattern that scales beyond centralized polling.

HLDEasy16 min readdistributedhealth-checksgossipfailure-detection

Requirements

What "alive" means and what we cut

Functional requirements

Non-functional requirements

Out of scope

Assumptions

Capacity Estimation

Heartbeat traffic, storage, server-count math

Active polling (centralized)

Passive push (nodes send to service)

Gossip (decentralized)

Storage

API Design

Push, query, subscribe

1. Heartbeat push (node → service)

POST /v1/heartbeats
Authorization: Bearer <node-token>

{
  "nodeId":   "pod-42",
  "version":  "v1.4.2",
  "metadata": { "region": "us-east-1", "zone": "1a" }
}

{ "ack": "ok", "nextHeartbeatBy": "2026-05-24T12:00:02Z" }

nextHeartbeatBy gives the node a deadline — if it can't push by then, it should retry urgently. Standard convention is 3× the configured interval.

2. Query

GET /v1/nodes/{nodeId}/status

{
  "nodeId":  "pod-42",
  "status":  "healthy",       // healthy | suspect | unhealthy | unknown
  "lastBeat": "2026-05-24T11:59:58.345Z",
  "uptime":  86400
}

GET /v1/nodes?status=unhealthy

Returns all unhealthy nodes — used by dashboards and on-call alerts.

3. Subscribe (events)

GET /v1/events/stream
Accept: text/event-stream

Server-Sent Events stream of status transitions:

event: node_unhealthy
data: { "nodeId": "pod-42", "lastBeat": "...", "missedBeats": 3 }

event: node_recovered
data: { "nodeId": "pod-42", "downtimeSec": 17 }

Failure semantics

Hysteresis prevents flapping.

High-Level Design

Centralized push vs gossip — full architecture diagrams

Option A — Centralized push

flowchart TB
  subgraph Nodes[Monitored fleet]
    N1[node-1]
    N2[node-2]
    Nn[node-N]
  end
  subgraph LB[Load balancer]
    L[LB]
  end
  subgraph HB[Heartbeat service]
    HB1[HB instance 1]
    HB2[HB instance 2]
    HBn[HB instance N]
  end
  subgraph Data[State store]
    Redis["(Redis cluster<br/>per-node state)"]
  end
  subgraph Sub[Subscribers]
    Alert[Alert pipeline]
    Disc[Service discovery]
    Dash[Dashboard]
  end

  N1 & N2 & Nn -->|"push every 1s"| L
  L --> HB1 & HB2 & HBn
  HB1 & HB2 & HBn --> Redis
  Redis -->|stream events| Alert & Disc & Dash

Simple to reason about. Single source of truth (Redis). Works up to ~100K nodes; beyond that, the central service becomes a bottleneck.

Option B — Gossip protocol (SWIM-style)

flowchart LR
  subgraph Cluster["Cluster — nodes know about each other"]
    N1[node-1]
    N2[node-2]
    N3[node-3]
    N4[node-4]
    N5[node-5]
  end
  N1 <-->|"gossip every 1s<br/>(3 random peers)"| N2
  N1 <-.->|gossip| N4
  N2 <-->|gossip| N5
  N3 <-->|gossip| N4
  N4 <-->|gossip| N5

Each node knows the membership list and chooses 3 random peers to exchange status updates with every 1s. Failure detection emerges from "I haven't heard about node X from anyone in 5s."

Pros:

Cons:

When to use which

Fleet size Recommendation
< 1K Centralized push — simplest
1K – 100K Centralized push + sharded service
> 100K Gossip (SWIM, Serf, Memberlist)

Hybrid

Production systems often combine both — gossip for in-cluster awareness + a central registry for outside-observers (dashboards, discovery). Each layer optimizes for what it's good at.

Database / State Design

Per-node state, time-windows, history retention

Per-node state (Redis hash)

HSET node:pod-42
  status          healthy
  lastBeat        1748098800123  # epoch ms
  consecutiveMisses 0
  region          us-east-1
  version         v1.4.2
EXPIRE node:pod-42 600  # auto-expire after 10 min of no beats

Key choice: Redis sorted set scored by lastBeat lets the failure-detector efficiently find "all nodes whose last beat was > 5s ago":

ZRANGEBYSCORE nodes-by-lastbeat -inf (now - 5000)

Status transition log (Cassandra / DDB)

{
  nodeId:    "pod-42",
  timestamp: 1748098800123,
  from:      "healthy",
  to:        "unhealthy",
  reason:    "5 missed beats",
  lastBeat:  "..."
}

Retained for 30 days for post-mortems. Cassandra wide-column or DynamoDB with TTL fits.

The failure-detector loop

Runs on a single HB instance (HA via leader election):

async function detectFailures() {
  const cutoff = Date.now() - 5000;
  const stale = await redis.zrangebyscore('nodes-by-lastbeat', '-inf', `(${cutoff})`);
  for (const nodeId of stale) {
    const cur = await redis.hget(`node:${nodeId}`, 'status');
    if (cur !== 'unhealthy') {
      await markUnhealthy(nodeId);
    }
  }
}
setInterval(detectFailures, 1000);

For 1M nodes this loop is O(log N) per scan since we only need the slice past the cutoff.

Deep Dive: Scaling

Sharding, gossip math, multi-region

Sharding the centralized service

flowchart TB
  Nodes[10K nodes] --> LB
  LB["Load balancer<br/>hash nodeId mod N"]
  LB --> S1["Shard 1<br/>nodes a-h"]
  LB --> S2["Shard 2<br/>nodes i-p"]
  LB --> S3["Shard 3<br/>nodes q-z"]
  S1 --> R1["(Redis)"]
  S2 --> R2["(Redis)"]
  S3 --> R3["(Redis)"]

Gossip math

In a cluster of N nodes, with each node gossiping to 3 random peers per round:

That's the worst case. In practice, exponential spread makes detection near-instant for the majority.

Multi-region

Bottlenecks & Trade-offs

Why this is harder than it looks

Top failure modes

1. Network partition mistaken for node failure The HB service can't reach node X; node X is fine but partitioned. We mark it unhealthy → orchestrator kills it. False positive. Mitigation: gossip provides multiple observation paths. If 5+ peers can see X, the central HB should trust them over its own probe.

2. HB service itself fails We need to monitor the monitor. Mitigation: HA pair with leader election (etcd, raft). Dead-man switch: an external observer (a separate process) alerts if the HB service stops emitting events.

3. Flapping under intermittent load Slow node misses beats randomly. Mitigation: hysteresis (5 misses to unhealthy, 3 successes to healthy). Slow-start on recovery.

4. Clock skew across nodes Node X's "now" disagrees with the HB service's "now." Mitigation: trust only the HB service's clock for "did we hear from X by T?". Don't compare node-side timestamps.

Accepted trade-offs

Trade-off Got Gave up
1s heartbeat interval Fast detection More traffic
5-miss threshold Low false positives 5s slower detection
Centralized at small scale Simple Bottleneck > 100K
Eventually consistent status Cross-region works Status can be 10s stale across regions

Monitoring (meta-monitoring)

Bottlenecks & Trade-offs

What we revisit at scale

At 100K nodes

At 1M nodes

At 10M+ nodes

Edge: monitoring IoT or edge devices

Different constraints — devices may be offline for hours then briefly online. The "unhealthy in 5s" semantic doesn't apply. Use lazy heartbeats with longer windows and intent-based status ("expected to be online during business hours").

What we'd integrate with

Interview: The centralized-vs-gossip trade-off is the discussion you want. Both work; the right answer depends on N. Strong candidates ask "how big is the fleet?" early.

Practice this question