Free lesson · System Design Advanced

Capacity Planning for Failure

Hard11 min readsystem-designadvanced

Here's something that took me years to internalize, and I see candidates get wrong in almost every system design interview at the staff level: failure is not an edge case. Failure is a capacity input.

If your capacity plan works perfectly when everything is running, you don't have a capacity plan. You have a sunny-day fantasy. Because in production, something is always broken. A host is rebooting. A disk is degraded. An AZ is having a bad hair day. A deploy is rolling through and half your fleet is on the new version while the other half is draining connections.

The question isn't "what if something fails?" The question is "which thing is failing right now, and is my system sized to handle it?"

The N+1, N+2, and N+AZ Models

When we talk about redundancy in capacity planning, we're really talking about how much extra capacity you keep idle so that when things break, the remaining healthy capacity can absorb the load without degrading.

N+1: The Minimum

N+1 means you have enough capacity to lose one unit (server, shard, node) and still serve all traffic within your SLO. If your system needs 10 servers to handle peak QPS at target P99 latency, you deploy 11.

This is the floor. It protects you against:

• A single host crash
• A single host reboot during patching
• A single disk failure
• One node in a cluster going unhealthy

It does not protect you against:

• An entire rack failure (which takes multiple hosts)
• An AZ disruption
• A correlated failure across hosts (bad deploy, shared dependency)

N+2: The Practical Minimum

N+2 means you can lose two units simultaneously and still serve traffic. Why two? Because in practice, failures overlap. A host crashes at 3 AM. The on-call pages but doesn't respond for 15 minutes. During those 15 minutes, the auto-scaler is trying to bring up a replacement. If a second host crashes before the replacement is ready - which happens more often than you'd think - N+1 is toast.

N+2 is what most production systems at scale actually run. It's the sweet spot between cost and resilience.

N+AZ: The Real Standard

Here's where it gets interesting. N+AZ means you can lose an entire Availability Zone and still serve all traffic from the remaining AZs.

graph TB
    subgraph "Region: us-east-1"
        subgraph "AZ-a (33% of fleet)"
            A1["Server 1"]
            A2["Server 2"]
            A3["Server 3"]
            A4["Server 4"]
        end
        subgraph "AZ-b (33% of fleet)"
            B1["Server 5"]
            B2["Server 6"]
            B3["Server 7"]
            B4["Server 8"]
        end
        subgraph "AZ-c (33% of fleet)"
            C1["Server 9"]
            C2["Server 10"]
            C3["Server 11"]
            C4["Server 12"]
        end
    end
    
    D["Load: needs 8 servers at peak"]
    E["N+AZ: deploy 12<br/>(lose any AZ, 8 remain)"]
    
    D --> E

This diagram shows a region with three Availability Zones, each containing 4 servers for a total of 12. The system needs 8 servers at peak load. If any single AZ goes down, the remaining two AZs have 8 servers, exactly meeting the requirement. This is N+AZ redundancy - sizing each AZ so the remaining AZs can absorb the full load.

If you have 3 AZs and you need N servers to handle peak load, N+AZ means deploying:

Total servers = N × 3/2 = 1.5N

Each AZ gets N/2 servers. If one AZ goes down, the remaining two AZs have 2 × (N/2) = N servers, which is exactly your peak requirement.

But wait - that leaves zero headroom after an AZ failure. In practice, you want headroom after the failure too:

Total servers = N × 3/2 × safety_margin

With a 1.2x safety margin post-failure: total = 1.8N.

Important: The AZ Rule of Thumb: For a 3-AZ deployment, N+AZ redundancy requires approximately 1.5x your peak capacity. With post-failure headroom, plan for 1.8x. This is the cost of AZ resilience - you're paying for nearly double the compute you "need" on a good day.

Blast Radius Analysis

"Blast radius" is the term we use for how much of your system a single failure can take out. It's a critical input to capacity planning because it determines how much spare capacity you need and where you need it.

Failure Domain Hierarchy

Failures don't all look the same. They have a hierarchy of blast radius, and your capacity plan needs to address each level:

Your capacity plan should explicitly address the first four. The last two are disaster recovery, which is a different (and more expensive) planning exercise.

Correlated Failures: The Sneaky One

The failure domain hierarchy above assumes independent failures. But in practice, failures are often correlated, and correlated failures have a much larger blast radius than their individual components suggest.

Here's what correlated failure looks like:

Deploy-induced correlation. You roll a bad deploy across your fleet. Even with canary deployments, if the canary passes (maybe the bug is load-dependent and the canary has low traffic), the full rollout might take out 30% of your fleet before you detect the problem and roll back.

Shared dependency correlation. Three of your services all depend on the same Redis cluster. When that cluster has a latency spike, all three services degrade simultaneously. Your capacity plan sized each service independently, but the failure is correlated.

Thundering herd on recovery. An AZ comes back online after a failure. All the clients that were retrying suddenly succeed at once. Your "recovered" AZ gets hit with a burst that's far above normal traffic. You sized for steady-state, not for recovery surge.

graph TD
    A["Redis Cluster<br/>Latency Spike 🔥"] --> B["Service A<br/>P99 degrades"]
    A --> C["Service B<br/>Timeouts start"]
    A --> D["Service C<br/>Queue depth rises"]
    
    B --> E["Service A callers<br/>retry 3x"]
    C --> F["Service B callers<br/>retry 3x"]
    D --> G["Service C callers<br/>retry 3x"]
    
    E --> H["Redis Cluster<br/>Load triples 🔥🔥🔥"]
    F --> H
    G --> H
    
    H --> I["Cascading failure<br/>across all three services"]

This diagram shows a correlated failure pattern. A Redis cluster experiences a latency spike, which causes three dependent services to degrade. Each service's callers begin retrying, tripling the load on the already-struggling Redis cluster. This creates a cascading failure across all three services. The blast radius of the initial Redis problem is amplified by correlated dependencies and retry behavior.

Warning: Retries amplify blast radius. If your services retry 3 times on timeout, a slowdown that affects 10% of your traffic can become a 30% load increase on the struggling dependency. Your capacity plan should account for retry amplification. The formula: effective_load = base_load × (1 + retry_rate × (retry_count - 1)). A 10% retry rate with 3 retries means 1.2x load. A 50% retry rate with 3 retries means 2x load.

Sizing for the Failure Scenarios That Matter

Let me give you a framework for deciding what to size for. Not every failure scenario deserves capacity. Some are rare enough that accepting degradation is the right trade-off.

The Severity × Frequency Matrix

quadrantChart
    title Failure Scenario Priority
    x-axis Low Frequency --> High Frequency
    y-axis Low Impact --> High Impact
    quadrant-1 Size For It
    quadrant-2 Must Size For It
    quadrant-3 Monitor Only
    quadrant-4 Auto-heal
    
    "AZ failure": [0.3, 0.85]
    "Host crash": [0.8, 0.2]
    "Bad deploy": [0.5, 0.7]
    "Region failure": [0.1, 0.95]
    "Process crash": [0.9, 0.1]
    "Dependency slow": [0.7, 0.6]

This quadrant chart maps failure scenarios by frequency and impact. The top-right quadrant, labeled Must Size For It, would contain high-frequency high-impact failures. The top-left, Size For It, includes AZ failure and region failure as low-frequency but high-impact events. Bad deploy and dependency slowdowns fall in the middle-high area. Host crashes and process crashes are high-frequency but low-impact, falling in the Auto-heal quadrant at the bottom-right. The bottom-left Monitor Only quadrant is for low-frequency, low-impact events.

Must size for (high frequency + high impact): These are rare in well-designed systems, but if you have one, it's your top priority.

Size for it (low frequency + high impact): AZ failures, bad deploys, major dependency outages. These don't happen often, but when they do, they take the service down if you haven't planned for them.

Auto-heal (high frequency + low impact): Host crashes, process restarts, minor network blips. These happen constantly but your system should auto-heal without human intervention. Size for N+2 and move on.

Monitor only (low frequency + low impact): Not worth explicit capacity planning. Accept graceful degradation.

The Production War Story

Early in my time at Amazon, we had a service that was sized perfectly for its peak traffic. We'd done the math, load tested it, felt confident. Then came a planned maintenance event where AWS was doing network maintenance in one AZ.

The maintenance was announced weeks in advance. We knew the AZ would be "impaired" for about 2 hours. Our N+AZ sizing should have handled it. But here's what we missed: the maintenance window overlapped with our weekly deployment window. So we had one AZ impaired by AWS maintenance, and our deployment system was rolling through the other two AZs simultaneously.

For about 8 minutes, our effective capacity was one AZ minus the hosts that were draining connections for the deploy. We dropped to roughly 40% of peak capacity during a period that was about 80% of peak traffic. P99 latency spiked to 4 seconds. Errors hit 12%.

The fix was a policy, not more servers: never deploy during known impairment events. But the lesson was broader - your capacity plan needs to account for overlapping failures, not just individual ones. The probability of two independent events overlapping is low. The probability of a planned event overlapping with another planned event is high, because humans batch maintenance.

Tip: The Overlap Rule: When planning capacity, list your top 3 most common failure/impairment scenarios and ask: "What if two of these happen at the same time?" If the answer is "we can't serve traffic," you're undersized - not for the individual events, but for the overlap.

Computing Your Redundancy Budget

Here's a practical approach to deciding how much redundancy to carry for each failure domain.

Step 1: List Your Failure Domains

For each service in your architecture, identify:

• What's the smallest unit that can fail? (process, container, host)
• What's the largest correlated failure? (AZ, region)
• What dependencies can cause correlated degradation?

Step 2: Assign a Redundancy Model

Step 3: Calculate the Cost

Here's the part that makes finance teams uncomfortable. For a Tier 0 service:

Redundancy cost = (1.8x - 1.0x) × base_cost = 0.8x base_cost

You're paying 80% more than the minimum to handle the worst case. But the alternative is downtime during an AZ failure, which for most revenue-generating services costs far more than the extra compute.

The framing that works for budget conversations: redundancy is not waste. It's capacity that's currently serving headroom and will serve traffic during failure. Those idle servers aren't doing nothing - they're your insurance policy.

Interview: The question interviewers love: "How do you decide the redundancy level for this system?"

Don't just say "N+2" or "N+AZ." Tie the redundancy model to the service's criticality and its failure impact. "For the payment processing path, I'd use N+AZ with 1.8x peak capacity because a payment outage has direct revenue impact and regulatory implications. For the analytics pipeline, N+2 is sufficient because a brief processing delay doesn't affect user experience."

That answer shows you think about trade-offs, not just formulas.

Graceful Degradation as a Capacity Strategy

Here's a perspective that changes how you think about sizing: you don't need enough capacity to serve 100% of features during a failure. You need enough capacity to serve the features that matter.

Graceful degradation means your system has a hierarchy of features it can shed as capacity decreases. Think of it like a building with floors - when the elevator breaks, you can still use the stairs. The building doesn't become unusable.

For a social media feed service, the degradation ladder might look like:

If you plan your capacity around 100% feature delivery during failure, you need massive redundancy. If you plan around serving the 75% tier during AZ failure, your redundancy budget drops significantly.

graph TD
    A["Normal: 100% capacity"] --> B{"Failure detected?"}
    B -->|No| A
    B -->|Yes| C{"Capacity remaining?"}
    C -->|"> 75%"| D["Shed: live counts, stories"]
    C -->|"50-75%"| E["Shed: recommendations"]
    C -->|"25-50%"| F["Shed: personalization"]
    C -->|"< 25%"| G["Static fallback page"]
    
    D --> H["Monitor + recover"]
    E --> H
    F --> H
    G --> H

This flowchart shows a graceful degradation decision tree. Under normal conditions, the system runs at 100% capacity. When a failure is detected, the system checks remaining capacity. Above 75%, it sheds live counts and stories. Between 50 and 75%, it disables the recommendations engine. Between 25 and 50%, all personalization is turned off. Below 25%, a static fallback page is served. At every degradation level, the system monitors and works toward recovery.

Tip: The Degradation Budget: Define your degradation tiers before the failure, not during it. Each tier should be load-tested independently. The worst time to discover that your "shed recommendations" mode has a bug is during the AZ failure when you first try to use it.

Six Takeaways

1. Failure is a capacity input, not an exception. If your capacity plan only works when everything is healthy, you don't have a capacity plan. Something is always failing in production.

2. N+AZ is the real standard. N+1 protects against a single host failure. N+AZ protects against the failure that actually wakes up the VP. For anything on the critical path, size for AZ loss.

3. Correlated failures have amplified blast radius. Retries, shared dependencies, and thundering herds on recovery all multiply the impact of a single failure. Account for the amplification, not just the initial event.

4. Overlapping failures are more likely than you think. Planned maintenance overlapping with a deploy. A dependency slowdown overlapping with a traffic spike. Size for the overlap of your top failure scenarios.

5. Redundancy is not waste - it's pre-positioned capacity. Frame it this way for budget conversations. Those "idle" servers are serving headroom now and will serve traffic during failure.

6. Graceful degradation reduces your redundancy budget. If you don't need to serve 100% of features during failure, you don't need to size for 100%. Define degradation tiers, test them, and use them to right-size your failure capacity.

Exercises

Exercise: Starter: You have a 3-AZ deployment with 30 servers (10 per AZ). Your peak load requires 24 servers. Can you survive an AZ failure? What if two hosts in a second AZ are also down for maintenance?

Exercise: Intermediate: Design a graceful degradation strategy for an e-commerce checkout service. Define at least 4 degradation tiers, specifying what features each tier sheds and what capacity level triggers the transition. Consider: product recommendations, real-time inventory checks, coupon validation, payment processing, order confirmation emails.

Exercise: Advanced: A service has 3 critical dependencies: a database (P99 latency 15ms), a cache (P99 latency 2ms), and an ML scoring service (P99 latency 50ms). Each caller retries up to 3 times with exponential backoff. Model the effective load on each dependency when the ML scoring service has a 30% timeout rate. How does retry amplification change your capacity plan for the ML service?

Back to System Design Advanced