AlgoInsight

Worked system design solution

Design an IoT Data Platform

An IoT data platform onboards millions of constrained devices, ingests their telemetry over MQTT, runs edge filtering to keep cloud costs sane, stores time-series data efficiently for years, maintains a queryable "shadow" state for each device, sends commands back to devices, manages firmware over-the-air, and detects anomalies in near real time. The design must work with intermittent connectivity (a sensor on a wind turbine may go offline for hours), constrained device memory (some have only 64 KB RAM), and security at scale (each device has unique credentials and can only access its own resources).

HLDHard28 min readiotmqtttime-seriesedge-computingdevice-management

Requirements

Device lifecycle, telemetry ingest, shadow state, commands, OTA

Functional Requirements

Non-Functional Requirements

Out of Scope

Assumptions

Capacity Estimation

Message rates, storage, MQTT broker fan-in, shadow store

Ingestion Rate

50 M devices × 1 msg/10s avg = 5 M msg/sec sustained.

Event Rate
Normal telemetry 5 M msg/sec
Event burst (5% of devices in trouble) 20 M msg/sec
Command responses 100 k/sec
Status/heartbeat 500 k/sec

Per message ~200 B → 1 GB/sec sustained ingest bandwidth.

Storage

5 M msg/sec × 200 B × 86400 sec/day = 86 TB/day raw.

After columnar compression (time-series formats achieve 10-20x compression on numeric data):

At cold-tier pricing ($5/TB/month object storage): cold tier costs $80k/month. Manageable.

Shadow Store

Each device has a shadow record:

Stored in DynamoDB or a sharded Redis cluster, RF=3 → 450 GB. Trivial.

MQTT Broker Capacity

A modern MQTT broker (e.g., HiveMQ, EMQ X, Mosquitto Cluster) handles ~1 M connections per broker node at ~100k msg/sec.

Edge Gateway

Many deployments use an edge gateway aggregating ~100 nearby devices:

Command Throughput

Most devices receive commands rarely (~1/hour during normal operation; bursts during firmware push).

Hot Query QPS

Time-series queries (dashboards):

Cached aggregates absorb most reads. Raw time-series storage is read primarily for export and ad-hoc investigation.

Note: The economics are unusual: each individual message is cheap to handle, but the volume is enormous. Engineering optimisations focus on cost-per-message: compression, batching, edge aggregation, tiered storage.

API Design

MQTT topics, REST control plane, shadow API, OTA flow

Device → Cloud: MQTT Topics

Devices publish telemetry on hierarchical topics:

devices/{device_id}/telemetry         # frequent data
devices/{device_id}/events            # discrete events (errors, state changes)
devices/{device_id}/status            # heartbeat: online / offline
devices/{device_id}/shadow/update     # device reports its state

Devices subscribe to receive commands:

devices/{device_id}/cmd               # commands from cloud to this specific device
devices/{device_id}/shadow/desired    # desired state changes

Topic ACLs restrict each device to publishing only under its own {device_id} prefix.

Cloud REST API — Device Management

POST /v1/devices
{
  "type": "thermostat-v2",
  "owner_id": "cust_acme",
  "tags": ["zone-eu", "model-2024"],
  "attributes": { "site": "Berlin-HQ" }
}
→ 201 {
    "device_id": "dev_abc123",
    "credentials": { "type": "x509", "cert_pem": "...", "key_pem": "..." }
  }

GET    /v1/devices/{device_id}
PATCH  /v1/devices/{device_id}                    # update tags, owner
DELETE /v1/devices/{device_id}                    # revoke credentials, remove
GET    /v1/devices?tag=zone-eu&fw_version=<2.4    # fleet search

Shadow API (cloud-side)

GET /v1/devices/{device_id}/shadow
→ 200 {
    "reported": { "temp_c": 22.4, "humidity": 0.42, "battery_pct": 87, "fw": "2.3.1" },
    "desired":  { "temp_c": 22.0 },
    "delta":    { "temp_c": -0.4 },
    "version":  42,
    "reported_at": "2026-05-26T18:00:00Z"
  }

POST /v1/devices/{device_id}/shadow/desired
{ "temp_c": 21.0 }
→ 202 { "version": 43, "delivery": "queued" }

When the device next connects, it gets the new desired state, applies it, and reports the new reported state. The cloud sees delta = empty once converged.

Command API

POST /v1/devices/{device_id}/commands
{
  "command": "reboot",
  "params": {},
  "ttl_seconds": 3600
}
→ 202 { "command_id": "cmd_xyz", "status": "queued" }

GET /v1/commands/{command_id}
→ 200 { "status": "delivered", "delivered_at": "...", "ack": { ... } }

If the device is offline, the command is queued for up to TTL seconds. If TTL expires before reconnection, the command is marked expired.

OTA API

POST /v1/firmware
{
  "device_type": "thermostat-v2",
  "version": "2.4.0",
  "image_url": "s3://...",
  "signature": "..."
}
→ 201 { "firmware_id": "fw_v240" }

POST /v1/rollouts
{
  "firmware_id": "fw_v240",
  "target_tag": "zone-eu",
  "strategy": "progressive",
  "stages": [
    { "percent": 1,   "wait_minutes": 60 },
    { "percent": 10,  "wait_minutes": 120 },
    { "percent": 50,  "wait_minutes": 360 },
    { "percent": 100 }
  ]
}
→ 201 { "rollout_id": "ro_42", "estimated_complete": "..." }

Time-Series Query API

POST /v1/timeseries/query
{
  "device_id": "dev_abc123",
  "fields": ["temp_c", "humidity"],
  "from": "2026-05-25T00:00:00Z",
  "to":   "2026-05-26T18:00:00Z",
  "aggregation": "avg",
  "interval": "5m"
}
→ 200 {
    "series": [
      { "field": "temp_c", "points": [{ "t": "...", "v": 22.1 }, ...] },
      { "field": "humidity", "points": [...] }
    ]
  }

High-Level Design

MQTT brokers, ingest pipeline, shadow service, time-series store, OTA

Architecture

flowchart TB
  subgraph EdgeTier["Edge / Field"]
    Device1[IoT Device]
    Device2[IoT Device]
    Gateway["Edge Gateway<br/>filters, aggregates,<br/>buffers offline"]
  end

  subgraph EdgeNet[Network Edge]
    MQTTLB["Network LB<br/>TCP / TLS"]
  end

  subgraph IngestPlane[Ingestion Plane]
    Broker1["MQTT Broker<br/>handles sub/pub<br/>auth, ACLs"]
    Broker2[MQTT Broker]
    BrokerN[MQTT Broker]
    Kafka["(Kafka<br/>telemetry-events<br/>commands-out)"]
  end

  subgraph ProcessingPlane[Processing Plane]
    EnrichSvc["Enrichment Pipeline<br/>Flink: parse, validate,<br/>add device metadata,<br/>route by tenant"]
    AnomalyDet["Anomaly Detection<br/>rolling stats per device"]
    AlertingSvc[Alerting]
  end

  subgraph StatePlane[State Plane]
    ShadowSvc["Shadow Service<br/>low-latency state"]
    ShadowStore["(Shadow Store<br/>DynamoDB / Redis)"]
    DevReg["(Device Registry<br/>DynamoDB)"]
  end

  subgraph DataPlane[Data Plane]
    TSDB["(Time-Series DB<br/>InfluxDB / TimescaleDB<br/>ClickHouse for analytics)"]
    ColdStore["(Cold Storage<br/>Parquet on S3<br/>5-year retention)"]
  end

  subgraph ControlPlane["Control / Mgmt Plane"]
    DevAPI[Device API]
    OTAOrchestrator[OTA Orchestrator]
    FirmwareCDN["Firmware CDN<br/>S3 + signed URLs"]
  end

  Device1 --> MQTTLB
  Device2 --> Gateway --> MQTTLB
  MQTTLB --> Broker1
  MQTTLB --> Broker2
  Broker1 --> Kafka
  Broker2 --> Kafka
  Kafka --> EnrichSvc
  EnrichSvc --> ShadowSvc --> ShadowStore
  EnrichSvc --> TSDB
  EnrichSvc --> AnomalyDet --> AlertingSvc
  TSDB -.->|age out| ColdStore
  DevAPI --> DevReg
  DevAPI --> ShadowSvc
  OTAOrchestrator --> Kafka
  Kafka -->|commands| Broker1
  Broker1 -->|publish to device topic| Device1
  Device1 --> FirmwareCDN

Ingestion Path

  1. Device opens a persistent MQTT over TLS connection to the load balancer.
  2. LB routes to an MQTT broker pod (sticky by device-id hash).
  3. Broker authenticates the device (client cert validated against device registry).
  4. Device publishes telemetry; broker pushes to Kafka topic telemetry-raw.
  5. Flink job consumes Kafka, validates schema, enriches with device metadata (tenant, location), routes to:
    • Shadow service: update last-reported state.
    • TSDB: persist for historical queries.
    • Anomaly detector: check against expected ranges.
  6. Anomaly detector publishes alerts to a separate Kafka topic; Alerting service notifies operators (PagerDuty, Slack, webhooks).

MQTT Broker Cluster

The broker tier is the most operationally sensitive part:

Brokers share state via bridging (each broker forwards messages on shared topics to peers). For "publish to a specific device" we look up which broker holds that device's connection (in Redis) and route accordingly.

Shadow Service

The shadow is a tiny but critical service. Read latency target: < 10 ms p99 — dashboards depend on it.

sequenceDiagram
  participant D as Device
  participant B as MQTT Broker
  participant K as Kafka
  participant F as Flink
  participant S as Shadow Service
  participant DB as Shadow Store

  D->>B: PUBLISH devices/d/shadow/update {"temp_c": 22.4}
  B->>K: telemetry-events
  K->>F: consume
  F->>S: updateReported(d, {"temp_c": 22.4})
  S->>DB: HSET shadow:d reported.temp_c 22.4
  S->>S: compute delta vs desired
  alt delta non-empty
    S->>K: command-out {device: d, set: {...}}
    K->>B: publish devices/d/shadow/desired
    B->>D: deliver
  end

OTA Orchestrator

Manages firmware rollouts as state machines:

Database Design

Device registry, shadow schema, TSDB layout, command store

Device Registry (DynamoDB)

PK: device_id
Attrs:
  type           string         # 'thermostat-v2'
  owner_id       string
  tags           list<string>
  attributes     map<string,string>
  fw_version     string
  hw_version     string
  cert_fingerprint string
  status         string         # 'active' | 'suspended' | 'retired'
  registered_at  timestamp
  last_connect   timestamp

GSI by owner_id for "list my devices" queries. GSI by fw_version for "find all devices on old firmware".

Shadow Store (DynamoDB or Redis)

PK: device_id
Attrs:
  reported       map (last reported by device)
  desired        map (target state set by cloud)
  delta          map (computed: desired - reported, if differs)
  version        number (incremented on each change)
  reported_at    timestamp
  desired_at     timestamp

Updates use conditional writes with version to prevent lost updates from concurrent reporters.

For sub-10ms reads we use a Redis cache hot tier (top 10% most-accessed devices live in Redis; cold devices read directly from DynamoDB).

Time-Series Database

Two-tier design:

Hot tier — TimescaleDB / InfluxDB or ClickHouse:

CREATE TABLE telemetry (
  ts          TIMESTAMPTZ NOT NULL,
  device_id   TEXT NOT NULL,
  tenant_id   TEXT NOT NULL,
  field       TEXT NOT NULL,
  value_num   DOUBLE PRECISION,
  value_text  TEXT,
  PRIMARY KEY (tenant_id, device_id, field, ts)
);
SELECT create_hypertable('telemetry', 'ts', chunk_time_interval => INTERVAL '1 hour');

Partitioned by hour for fast prune. Per-device queries hit a small number of partitions.

Cold tier — Parquet on S3, partitioned by year=/month=/day=/tenant=. Queried via Athena / DuckDB for long-term analytics.

Aged-out data moves from hot to cold daily via a Spark job.

Pre-Aggregated Rollups

Storing 1-Hz telemetry for 5 years is expensive. We pre-aggregate:

A dashboard query for "last 30 days" hits the 1-min table, not the raw table. 30 days × 1440 min = 43,200 points per field — fast to query, easy to plot.

Command Store

PK: command_id
Attrs:
  device_id   string
  command     string
  params      json
  status      string    # 'queued' | 'delivered' | 'acked' | 'expired' | 'failed'
  created_at  timestamp
  delivered_at timestamp
  acked_at    timestamp
  ack_payload json
  ttl_at      timestamp

TTL field auto-expires undelivered commands. A status index supports "list active commands for device X".

Rollout State

PK: rollout_id
Attrs:
  firmware_id   string
  target_filter map
  current_stage int
  stages        list<{ percent, wait_minutes, status }>
  total_devices int
  succeeded     int
  failed        int
  status        string  # 'running' | 'paused' | 'completed' | 'aborted'

Per-device rollout assignments:

PK: rollout_id, SK: device_id
Attrs: status, attempts, last_error, completed_at

Deep Dive

MQTT semantics, edge filtering, anomaly detection, OTA safety

MQTT QoS Levels

MQTT defines three Quality of Service levels:

QoS Guarantee Use case Overhead
0 At most once High-frequency telemetry where loss is OK Fire and forget
1 At least once Default for most telemetry PUBACK round trip
2 Exactly once Commands, financial events 4-step handshake

For 99% of telemetry we use QoS 1 — the cost of duplicates (handled by idempotent message-id deduplication downstream) is far less than the cost of QoS 2's extra round-trips. Critical commands use QoS 2.

Edge Filtering

A gateway sitting between a fleet of devices and the cloud is a money-saver:

Edge software is configurable from the cloud and updated like firmware. Typical bandwidth reduction: 10-50x.

Anomaly Detection

For each device + field, maintain a rolling baseline:

Score each new reading:

False positives are inevitable. We tier alerts:

OTA Safety

Bricking 50 M devices is a company-ending event. Defences:

  1. A/B partitions on the device: firmware installs to inactive partition; reboot into it; if device fails to "phone home" within 5 minutes, auto-rollback to the previous partition.
  2. Canary stages: rollouts proceed in increments (1% → 10% → 50% → 100%). If failure rate breaches threshold, auto-pause.
  3. Signature verification: firmware signed with HSM-protected key; device verifies signature before installing.
  4. Cohort segmentation: deploy to dev/test fleets first, then internal devices, then external — never deploy to 100% in one wave.
  5. Operator kill switch: a single API call halts an in-progress rollout globally.

Device Authentication

Two common schemes:

X.509 mutual TLS: each device has a unique cert signed by our CA. Pros: standard, hardware-friendly (TPM-backed keys). Cons: cert management at scale.

Pre-shared key: device knows a symmetric secret. Pros: simpler. Cons: rotation is painful; if leaked from device firmware, fleet-wide breach.

We use X.509 with intermediate CAs per fleet. A revocation list (CRL) is published every 5 minutes; brokers reload and reject revoked certs.

Backpressure Handling

What happens when ingestion can't keep up? Options:

Our SDK supports local buffering of up to 1 hour of data per device. Acks slow down ⇒ devices buffer ⇒ when cloud recovers, devices flush. Bandwidth spike during recovery but no data loss.

Connection Storms

After a network partition heals, millions of devices reconnect simultaneously — connection storm. Defences:

Time Synchronization

Many IoT use cases need accurate timestamps (industrial monitoring, traffic). Devices use:

Scaling

Sharding, regional brokers, TSDB partitioning, multi-tenant isolation

Sharding Strategy

flowchart TB
  subgraph Devices[Devices]
    D1[d_001..d_999]
    D2[d_1000..d_1999]
    D3[d_2000..d_2999]
    DN[...]
  end
  Router["Connection Router<br/>hash(device_id)"]
  subgraph BrokerShards[Broker Shards]
    BS1["Broker Shard 1<br/>connections + topics"]
    BS2[Broker Shard 2]
    BS3[Broker Shard 3]
    BSN[Broker Shard N]
  end
  subgraph KafkaPart[Kafka Partitions]
    KP1[Partition 1]
    KP2[Partition 2]
    KPN[Partition N]
  end

  D1 --> Router --> BS1
  D2 --> Router --> BS2
  D3 --> Router --> BS3
  DN --> Router --> BSN
  BS1 --> KP1
  BS2 --> KP2
  BS3 --> KP3
  BSN --> KPN

Devices are sharded by hash(device_id). The hash determines:

Co-location of these decisions enables single-shard processing — no cross-shard joins needed for shadow updates.

Regional Deployment

For latency and data residency:

Each region is independent: its own MQTT brokers, Kafka, Shadow Store, TSDB. Devices connect to the regional endpoint closest to them, configured at provisioning.

Cross-region replication is optional — only if customer needs global dashboards. By default data stays in-region.

Auto-Scaling the Broker Tier

Brokers are stateful (connection + topic state). Scaling rules:

Migration of devices from one broker to another: tell the device to reconnect to a new endpoint (via MQTT 5.0 SERVER_REDIRECT packet). Migration completes in seconds with no data loss.

Auto-Scaling the Time-Series Tier

TSDB is sharded by tenant + hash(device_id). Hot shards become hotspots when a single tenant has 10 M devices.

Multi-Tenant Isolation

Tenants must not see each other's data. Enforcement:

For very-large or compliance-sensitive customers we deploy dedicated cells: physically separate broker cluster, Kafka, TSDB.

Cold Storage Lifecycle

After 7 days in TSDB hot tier:

Queries spanning hot + cold tier are seamless: a federated query engine (e.g., Trino) joins TSDB + S3.

Capacity Headroom

Bottlenecks & Trade-offs

Connection scaling, ingest cost, OTA blast radius, anomaly false positives

Trade-off Summary

Decision Choice Alternative Why
Protocol MQTT HTTP, CoAP Most battery-efficient; pub/sub native
Ingestion buffer Kafka Direct to TSDB Decouple ingest from storage; absorb bursts
Shadow store Per-region DynamoDB + Redis cache Global Spanner Per-region OK for shadows; cost wins
TSDB Tiered (hot DB + cold S3) Hot only 90% of queries are recent; cold storage 10× cheaper
QoS Mostly QoS 1 QoS 2 Speed wins; idempotent downstream handles duplicates
OTA Progressive rollout with auto-pause Big-bang Safety; 50M devices is too many to brick
Anomaly detection Per-device rolling stats Global ML model Per-device adapts to each device's normal

Bottlenecks

1. Broker connection state replication When devices reconnect simultaneously (network event), brokers thrash on session-state lookups in Redis. Mitigations: shard Redis by device hash; pre-warm shards before known events.

2. Kafka consumer lag during bursts A 4x telemetry burst can lag Flink consumers by minutes. Mitigations: scale Flink ahead of known patterns; rate-limit per device at the broker to prevent runaway devices.

3. Time-series write amplification Per-device per-field per-timestamp = many small rows. TimescaleDB compresses by chunk only after a delay. Cost-per-message stays high in the hot tier. Mitigations: ClickHouse for analytics queries; Parquet for cold storage.

4. OTA blast radius A bad firmware can degrade 50M devices' performance instantly even before bricking. Mitigations: phased rollout, A/B install, kill switch.

5. Anomaly false positives Operators get alarm fatigue. Mitigations: per-tenant tuning, alarm grouping, severity scoring.

Trade-offs Made

What Would Break First as Traffic Grows

  1. Broker connection state in Redis at 500 M devices — would shard Redis more aggressively.
  2. Kafka topic count when each customer wants their own topic — switch to single topic with header-based routing.
  3. OTA orchestration at very large rollouts (50 M devices) — would parallelise more aggressively, partition by region.
  4. Shadow store contention on hot shadows (devices reporting 100x/sec) — would partition the shadow itself.

Failure Scenarios

Failure Behaviour Recovery
Single broker pod down ~1 M devices reconnect to another broker Auto in seconds
Region outage Devices fail to connect; buffer locally; reconnect when region recovers Hours-scale; data preserved
TSDB write failure Kafka holds messages; Flink retries Auto when TSDB recovers
Shadow store down Reads from Redis cache (stale OK); writes queue Auto on recovery
OTA bad firmware Auto-rollback per device; orchestrator pauses fleet <1 hour to halt; per-device recovery

Future Improvements

Practice this question