Free lesson · System Design Advanced

Cardinality Explosion and Mitigation

Hard11 min readsystem-designadvanced

A developer adds a new metric to track API response times. Seems harmless:

http_request_duration_seconds{method="GET", endpoint="/api/users", status="200"}

Three labels, each with a handful of values. Maybe 50 unique time series. Prometheus handles that without blinking.

Then someone adds user_id as a label:

http_request_duration_seconds{method="GET", endpoint="/api/users", status="200", user_id="abc123"}

Your platform has 10 million users. Each unique combination of labels creates a separate time series. That single metric now generates 10 million time series. Multiply by the scrape interval (every 15 seconds), and you're writing 40 million data points per minute for one metric. Your Prometheus instance runs out of memory. Your metrics storage bill goes from $2,000/month to $200,000/month. Your dashboards time out.

This is cardinality explosion - the most common and most expensive mistake in metrics engineering. It's the reason observability at scale is hard, and it's the first thing you need to understand before designing any metrics system in a system design interview.

What Cardinality Means

Cardinality is the number of unique time series a metric produces. It's determined by the number of unique combinations of label values.

graph TD
    subgraph "Cardinality Calculation"
        METRIC["Metric: http_request_duration_seconds"]
        
        METRIC --> L1["Label: method<br/>Values: GET, POST, PUT, DELETE<br/>Cardinality: 4"]
        METRIC --> L2["Label: endpoint<br/>Values: /users, /orders, /products,<br/>/search, /auth<br/>Cardinality: 5"]
        METRIC --> L3["Label: status<br/>Values: 200, 201, 400, 404, 500<br/>Cardinality: 5"]
        
        SAFE["Total cardinality:<br/>4 × 5 × 5 = 100 time series<br/><br/>✅ This is fine.<br/>100 series per metric is normal."]
        
        L1 --> SAFE
        L2 --> SAFE
        L3 --> SAFE
    end
    
    subgraph "Cardinality Explosion"
        METRIC2["Same metric + user_id label"]
        
        METRIC2 --> L4["Label: user_id<br/>Values: 10,000,000 unique users<br/>Cardinality: 10,000,000"]
        
        EXPLOSION["Total cardinality:<br/>4 × 5 × 5 × 10,000,000<br/>= 1,000,000,000 time series<br/><br/>❌ ONE BILLION series.<br/>No metrics backend survives this."]
        
        L4 --> EXPLOSION
    end

This diagram shows cardinality calculation. A metric with three labels - method with 4 values, endpoint with 5 values, and status with 5 values - produces 100 unique time series, which is perfectly fine. Adding a user_id label with 10 million unique values multiplies the total to 1 billion time series. No metrics backend can handle this - this is cardinality explosion.

The Cost of Cardinality

Every time series consumes resources at every stage of the metrics pipeline:

Warning: Managed observability pricing is based on cardinality. Datadog charges per custom metric (unique time series). New Relic charges per data point ingested. Grafana Cloud charges per active series. A single bad label - request_id, user_id, session_id, trace_id - added to a high-traffic metric can generate a bill larger than your entire cloud infrastructure cost. This isn't theoretical. I've seen teams get surprise bills of $500,000+ from a single cardinality mistake deployed on a Friday and caught on Monday.

The Labels That Kill You

graph TD
    subgraph "Safe Labels (Bounded Cardinality)"
        SAFE_LABELS["✅ Labels with known, small value sets<br/>─────────────────────<br/>method: GET, POST, PUT, DELETE, PATCH (5)<br/>status_code: 200, 201, 400, 404, 500, 503 (~10)<br/>region: us-east-1, eu-west-1, ap-southeast-1 (~5)<br/>service: auth, orders, payments, search (~20)<br/>environment: prod, staging, dev (3)<br/>instance_type: c5.xlarge, m5.2xlarge (~10)<br/><br/>These multiply into thousands, not millions"]
    end
    
    subgraph "Dangerous Labels (Unbounded Cardinality)"
        DANGER_LABELS["❌ Labels with unbounded or huge value sets<br/>─────────────────────<br/>user_id: millions of unique values<br/>request_id / trace_id: unique per request<br/>email: unique per user<br/>ip_address: millions of unique values<br/>url_path with parameters: /users/12345/orders/67890<br/>error_message: free-text, thousands of variants<br/>pod_name in autoscaling: pod-xyz-abc123<br/>git_commit_sha: unique per deploy<br/><br/>Any one of these creates millions of series"]
    end

This diagram contrasts safe and dangerous metric labels. Safe labels have small, bounded value sets - method has 5 values, status code has about 10, region has about 5. These multiply into thousands of series, which is manageable. Dangerous labels have unbounded or huge value sets - user_id has millions, request_id is unique per request, URL paths with parameters create millions of combinations. Any single dangerous label creates millions of time series.

Tip: The rule of thumb: if a label's value set can grow with traffic or users, it doesn't belong on a metric. Metric labels should represent dimensions of the system (regions, services, status codes) not dimensions of the data (user IDs, request IDs, individual URLs). User-level and request-level data belongs in logs and traces, not metrics.

Mitigation Strategies

Strategy 1: Aggregation - Roll Up Before You Store

Instead of tracking per-user latency as a metric, aggregate:

graph TD
    subgraph "Before: Per-User Metric (Exploding)"
        RAW["http_latency{user_id='abc'} = 120ms<br/>http_latency{user_id='def'} = 85ms<br/>http_latency{user_id='ghi'} = 200ms<br/>... × 10 million users<br/><br/>10,000,000 time series"]
    end
    
    subgraph "After: Aggregated Metric (Bounded)"
        AGG["http_latency_bucket{le='50'} = 1,200,000<br/>http_latency_bucket{le='100'} = 3,500,000<br/>http_latency_bucket{le='250'} = 4,800,000<br/>http_latency_bucket{le='500'} = 4,950,000<br/>http_latency_bucket{le='+Inf'} = 5,000,000<br/><br/>5 time series (histogram buckets)<br/>Captures the DISTRIBUTION without<br/>individual user identity"]
    end
    
    TRADE["Trade-off:<br/>You can answer 'what's the p99 latency?'<br/>but NOT 'what was user abc's latency?'<br/><br/>For user-level queries → use logs/traces"]

This diagram shows aggregation as a cardinality mitigation strategy. Before aggregation, per-user latency metrics create 10 million time series. After aggregation into histogram buckets, the same latency distribution is captured in just 5 time series - one per bucket boundary. The trade-off is that you can answer distribution questions like p99 latency but cannot query individual user latency from metrics. User-level queries should use logs or traces instead.

Strategy 2: Label Allowlisting - Reject Before Ingestion

Configure your metrics pipeline to reject or drop labels not on an approved list:

graph TD
    APP["Application emits metric:<br/>http_requests{method='GET',<br/>endpoint='/users',<br/>user_id='abc123',<br/>request_id='req-456'}"]
    
    APP --> FILTER["Metrics Relay / Pipeline<br/>(e.g., Prometheus relabel_configs,<br/>Datadog Agent, Vector)<br/>──────────────────<br/>Allowlist: method, endpoint,<br/>status, region, service<br/><br/>Action: DROP user_id, request_id"]
    
    FILTER --> STORED["Stored metric:<br/>http_requests{method='GET',<br/>endpoint='/users'}<br/><br/>Cardinality: controlled"]

This diagram shows label allowlisting. The application emits a metric with four labels including user_id and request_id. The metrics relay filters against an allowlist of approved labels - method, endpoint, status, region, service. The dangerous user_id and request_id labels are dropped before ingestion, and only the allowed labels are stored, keeping cardinality controlled.

Strategy 3: Sampling - Keep a Fraction, Represent the Whole

For high-cardinality data you still need, sample rather than collect everything:

Strategy 4: Exemplars - Link Metrics to Traces

The problem with aggregation: you know the p99 latency spiked, but you don't know which requests were slow. Exemplars solve this by attaching a few sample trace IDs to each metric data point.

graph TD
    subgraph "Exemplars Bridge Metrics and Traces"
        METRIC_POINT["Metric: http_latency_bucket{le='500'}<br/>Value: 4,950,000<br/>Timestamp: 2024-01-15T14:30:00Z"]
        
        METRIC_POINT --> EXEMPLAR["Exemplar attached:<br/>trace_id=abc123def456<br/>value=487ms<br/><br/>This is ONE specific request<br/>that fell into this bucket.<br/>Click through to see the<br/>full distributed trace."]
        
        EXEMPLAR --> TRACE["Full distributed trace<br/>in Jaeger / Tempo / X-Ray<br/>──────────────────<br/>Shows exactly which service<br/>was slow and why"]
    end

This diagram shows exemplars bridging metrics and traces. A histogram metric data point tracks 4.95 million requests in the 500ms bucket. An exemplar is attached - a specific trace ID with a 487ms latency value. Clicking through the trace ID opens the full distributed trace in Jaeger or Tempo, showing exactly which service was slow and why. Exemplars let you keep aggregated metrics for cardinality control while still drilling into specific requests.

Important: Exemplars are the correct answer to "I need user-level data in metrics." When someone asks "can we add user_id to this metric so we can debug slow requests for specific users?" the answer is: "No - that would explode cardinality. Instead, we'll use exemplars to link the metric to a trace, and the trace contains the user_id. You get the same debugging capability without the cardinality cost."

Monitoring the Monitor: Cardinality Observability

graph TD
    subgraph "Cardinality Monitoring Pipeline"
        SCRAPE["Prometheus scrapes targets"] --> INDEX["TSDB ingests new series<br/><br/>Track: active series count<br/>per metric name,<br/>per label name"]
        
        INDEX --> ALERT["Alert rules:<br/>──────────────────<br/>1. Total active series > threshold<br/>   (warn at 5M, critical at 10M)<br/><br/>2. Single metric cardinality spike<br/>   (> 100K series for one metric)<br/><br/>3. New label appears on metric<br/>   that wasn't in allowlist<br/><br/>4. Cardinality growth rate<br/>   (> 10% increase in 1 hour)"]
        
        ALERT --> ACTION["Response:<br/>──────────────────<br/>1. Identify the metric<br/>   (prometheus_tsdb_head_series)<br/><br/>2. Find the offending label<br/>   (topk by label cardinality)<br/><br/>3. Drop or aggregate at relay<br/><br/>4. Fix the application code"]
    end

This diagram shows cardinality monitoring for the monitoring system itself. Prometheus scrapes targets and the TSDB ingests new series. The system tracks active series count per metric and per label. Alert rules fire when total series exceeds thresholds, when a single metric has a cardinality spike, when unexpected labels appear, or when cardinality grows rapidly. The response is to identify the metric, find the offending label, drop or aggregate at the relay layer, and fix the application code.

Exemplars: Getting Detail Without Cardinality

The newest mitigation strategy is exemplars - a feature in Prometheus and OpenTelemetry that lets you attach a single trace ID to a metric data point without increasing cardinality. Instead of creating a high-cardinality label like user_id on every metric (which creates millions of series), you record one exemplar per bucket with a link to a specific trace.

When you see a latency spike in a histogram, you click the exemplar and jump directly to the trace that caused that data point. You get the investigative depth of high-cardinality labels without the storage cost. The metric stays low-cardinality (aggregated). The trace provides the per-request detail. The exemplar is the bridge between them.

This is the modern answer to "I need per-request visibility but can't afford per-request cardinality": keep metrics aggregated, keep traces sampled, and use exemplars to connect the two.

Interview: "Your Prometheus just OOM'd. What happened and how do you prevent it?" This is a production maturity question. The strong answer: "The most common cause is cardinality explosion - a metric with an unbounded label like raw URL path, user ID, or request ID. Each unique label combination creates a new time series, and Prometheus stores all active series in memory. A single metric with a million unique label values creates a million series, consuming gigabytes of RAM. Prevention has three layers: first, label allowlists that reject unknown labels at the ingestion pipeline before they reach Prometheus. Second, relabeling rules that drop or aggregate high-cardinality labels at scrape time. Third, monitoring the monitoring - alerting on prometheus_tsdb_head_series so you catch cardinality growth before it causes an OOM. For deep investigation without high cardinality, I'd use exemplars to link aggregated metrics to individual traces."

The Production War Story

At Amazon, a team deployed a new service that emitted a metric called api_request_count with a label path. The label value was the raw URL path - including path parameters. Requests to /users/{userId}/orders/{orderId} generated labels like /users/abc123/orders/def456, /users/xyz789/orders/ghi012, and so on.

Within 6 hours, the metric had created 14 million unique time series. The team's Prometheus instance (using Cortex as the remote storage backend) ran out of memory and crashed. Worse, the crash happened during a separate production incident, so the team lost metrics visibility at the exact moment they needed it most.

The recovery was a three-step process: (1) identify the offending metric using Cortex's cardinality analysis API, (2) deploy a relabelling rule in the Prometheus config to strip path parameters - rewriting /users/abc123/orders/def456 to /users/{id}/orders/{id} - and (3) purge the 14 million series from storage (which took 4 hours because TSDB compaction had to process the deletions).

After this incident, we implemented a cardinality guard - a sidecar that runs alongside every Prometheus instance, monitors the active series count per metric, and automatically drops any metric that exceeds 50,000 series until an engineer explicitly approves it. That guard has caught 23 cardinality explosions in the two years since, none of which reached production impact.

The lesson: cardinality explosions happen fast - one bad deployment can create millions of series in minutes. Prevention (allowlists, guardrails, code reviews for metric labels) is orders of magnitude cheaper than recovery (purging millions of series from storage while your monitoring is down).

Six Takeaways

1. Cardinality is the product of all label value counts. A metric with four labels of 5, 10, 5, and 10,000,000 values produces 2.5 billion time series. One unbounded label destroys the system.

2. User IDs, request IDs, and free-text labels never belong on metrics. Metrics labels should represent system dimensions (regions, services, status codes), not data dimensions (users, requests, sessions). Per-entity data belongs in logs and traces.

3. Aggregation is the primary mitigation. Histograms capture distributions in bounded series. Counters with bounded labels capture rates. You lose individual-entity visibility in metrics but gain it back through traces.

4. Exemplars bridge metrics and traces. Attach sample trace IDs to metric data points so you can drill from an aggregated p99 spike into the specific slow trace - without adding trace IDs as metric labels.

5. Monitor your monitoring. Track active series count, per-metric cardinality, and cardinality growth rate. Alert on spikes. Automate guardrails that drop runaway metrics before they crash your monitoring infrastructure.

6. Managed observability pricing amplifies the problem. A cardinality mistake that's an engineering problem with self-hosted Prometheus becomes a six-figure financial problem with Datadog. Understand your vendor's pricing model before deploying metrics.

Exercises

Exercise: Starter: Your API has 8 endpoints, 5 HTTP methods, 6 status codes, and you deploy across 3 regions. (a) What's the cardinality of a metric with labels endpoint, method, status, region? (b) A developer wants to add a customer_tier label with values "free", "pro", "enterprise". What's the new cardinality? Is this safe? (c) Another developer wants to add request_id (unique per request, 1 million requests/hour). What's the cardinality now? (d) How would you redesign the metric to capture per-request data without cardinality explosion?

Exercise: Intermediate: You manage a metrics pipeline for a platform with 500 microservices, each emitting 20 metrics with an average of 200 time series per metric. (a) What's the total active series count? (b) Your Prometheus instance can handle 5 million active series. Are you within budget? (c) Three services have a cardinality bug - each emitting 500,000 series from a pod_name label in an autoscaling group. What's your new total? Is the Prometheus instance still healthy? (d) Design the cardinality guardrail: where in the pipeline do you add it (application, collector, relay, or storage), what thresholds do you set, and what action does it take when triggered? (e) You use Datadog at $0.05 per custom metric per month. What's your monthly bill at baseline? After the cardinality bug?

Exercise: Advanced: Design the metrics pipeline for a cloud provider with 50,000 customer accounts, each running 10-100 services, generating 2 billion metric data points per minute. (a) What's the estimated total active time series? State your assumptions. (b) Design the multi-tier storage: hot (real-time queries, last 2 hours), warm (dashboards, last 30 days), cold (compliance, 1 year). What storage technology for each tier? (c) A customer deploys a service that emits a metric with their database's primary key as a label - 100 million unique values. Your pipeline must handle this without affecting other customers. Design the per-customer cardinality limit with isolation. (d) You want to offer an "exemplars" feature: customers can attach trace IDs to metrics. Design the storage: where do exemplars live (in the metrics TSDB or a separate store), how do you limit exemplar cardinality, and how does the query path work (customer queries a metric, clicks an exemplar, sees the trace)?

Back to System Design Advanced