Free lesson · System Design Advanced

Multi-Region Capacity

Hard12 min readsystem-designadvanced

Everything we've talked about so far - P99 sizing, AZ failure planning, blast radius - has been within a single region. For a lot of systems, that's sufficient. You pick a region close to your users, spread across AZs, and call it done.

But the moment your system needs to be available when an entire region goes down, or the moment your users are spread across continents and 200ms of cross-ocean latency becomes unacceptable, you enter multi-region territory. And multi-region capacity planning is a fundamentally different game.

It's different because you're no longer just adding more servers. You're making architectural decisions about where data lives, which copy of the data is authoritative, and what happens when the two regions disagree. These decisions determine your capacity plan more than any QPS estimate ever will.

Active-Passive vs Active-Active: The Fundamental Fork

Before you can size a multi-region system, you need to decide which model you're running. This decision cascades into every capacity number you'll compute.

Active-Passive

One region handles all traffic. The second region sits warm (or cold), ready to take over if the primary fails.

graph LR
    subgraph "Primary Region (us-east-1)"
        A["Full Fleet<br/>100% traffic"]
        B["Primary Database"]
    end
    
    subgraph "Standby Region (us-west-2)"
        C["Warm Fleet<br/>0% traffic"]
        D["Replica Database<br/>Async replication"]
    end
    
    A --> |"Async replication"| D
    B --> |"Async replication"| D
    
    E["Users"] --> A
    E -.->|"Only during failover"| C

This diagram shows an active-passive multi-region setup. The primary region us-east-1 has a full fleet handling 100% of traffic and the primary database. The standby region us-west-2 has a warm fleet receiving 0% traffic and a replica database receiving asynchronous replication from the primary. Users normally connect only to the primary region and would be routed to the standby only during a failover event.

Capacity math for active-passive:

Primary region: sized for 100% peak + N+AZ redundancy = ~1.8x peak
Standby region: sized for 100% peak (must handle everything during failover)
Total: ~2.8x peak capacity
But the standby can be smaller instances, spot, or reserved at lower commitment
Realistic total: 2.0-2.5x peak

The catch: your standby region needs to handle 100% of traffic the instant the primary fails, but it's been sitting idle. Cold caches. Cold connection pools. Possibly stale DNS caches on the client side. The first 5-10 minutes after failover are the most dangerous, because the standby is absorbing full production traffic with zero warm-up.

Warning: The Cold Start Trap: Your standby region's stated capacity and its actual capacity on failover are different numbers. A warm cache might handle 10x the QPS of a cold cache for the same hardware. If you size your standby based on warm-cache performance numbers, you'll have a bad time during actual failover. Either keep the standby warm with synthetic traffic, or size it for cold-cache performance.

Active-Active

Both regions serve traffic simultaneously. Each region handles a portion of the load, typically based on geographic proximity.

graph LR
    subgraph "Region A (us-east-1)"
        A["Fleet A<br/>60% traffic"]
        B["Database A<br/>Multi-leader"]
    end
    
    subgraph "Region B (eu-west-1)"
        C["Fleet B<br/>40% traffic"]
        D["Database B<br/>Multi-leader"]
    end
    
    B <--> |"Bidirectional<br/>replication"| D
    
    E["US Users"] --> A
    F["EU Users"] --> C
    
    A -.->|"Failover overflow"| C
    C -.->|"Failover overflow"| A

This diagram shows an active-active multi-region setup. Region A in us-east-1 handles 60% of traffic from US users, while Region B in eu-west-1 handles 40% from EU users. Both regions have their own database using multi-leader replication with bidirectional synchronization. During failover, traffic can overflow from one region to the other, shown by dotted lines.

Capacity math for active-active:

This is where it gets subtle. If Region A handles 60% of traffic and Region B handles 40%, how do you size for a region failure?

Option 1: Full failover capacity in each region

Region A: sized for 100% peak (can absorb Region B's traffic)
Region B: sized for 100% peak (can absorb Region A's traffic)
Total: 2.0x peak capacity

This is the safest but most expensive. You're running 2x capacity and normally using less than half.

Option 2: Partial failover with degradation

Region A: sized for 80% peak (can absorb most of Region B's traffic, with degradation)
Region B: sized for 80% peak (can absorb most of Region A's traffic, with degradation)
Total: 1.6x peak capacity

During a region failure, the surviving region runs at ~100/80 = 125% of its normal capacity. Degradation tiers kick in (from lesson A1.2) to shed non-critical features. This is what most mature systems actually do.

Option 3: Geographic isolation (no cross-region failover)

Region A: sized for its own traffic with N+AZ redundancy = ~1.8x its share
Region B: sized for its own traffic with N+AZ redundancy = ~1.8x its share
Total: 1.8x peak capacity
But: no region failover. If Region A goes down, US users are down.

This is cheapest but provides no region-level resilience. Some systems use this intentionally - if your product is regional (a local marketplace, for example), cross-region failover adds cost without matching value.

The Traffic Split Problem

In active-active, how you split traffic between regions is itself a capacity planning decision. The three main approaches each have different implications:

Geographic Routing

Route users to the nearest region based on their location. US users go to us-east-1, EU users go to eu-west-1. This is the most common approach and usually the right default.

Capacity implication: Your traffic split is determined by your user geography. If 70% of your users are in the US, your US region handles 70% of traffic. This is inherently unbalanced, which means your failover capacity calculation is asymmetric - the US region needs less relative headroom (it's already large) while the EU region needs more (it must absorb a larger relative increase during US failure).

Latency-Based Routing

Route users to the region that responds fastest from their location, measured dynamically. This handles edge cases that geographic routing misses - a user in Brazil might get lower latency to us-east-1 than to sa-east-1, depending on network topology.

Capacity implication: Similar to geographic, but the split is less predictable. You need more headroom because the split can shift based on network conditions.

Load-Based Routing

Route users to the region with more available capacity, regardless of geography. This actively balances load across regions, which simplifies capacity planning but adds latency for users routed to the distant region.

Capacity implication: The most even split, which means the most efficient capacity utilization. But users pay for it in latency. Cross-ocean round trips add 60-150ms per request, which may violate your latency SLO for some percentage of users.

Tip: The Practical Answer: Start with geographic routing. It gives the best user experience and the most predictable capacity planning. Add load-based overflow only for failover scenarios - when a region is impaired, shift its traffic to the surviving region. Don't route healthy traffic to distant regions just for load balancing.

Cross-Region Data Replication: The Hidden Capacity Cost

Here's the capacity planning angle that most people miss when they think about multi-region: replication bandwidth is a real cost, and it grows with your write volume.

In active-active, every write in Region A must be replicated to Region B, and vice versa. The replication bandwidth is:

Replication bandwidth = write_QPS × average_write_size × replication_factor

For a system doing 10,000 writes/second with an average write size of 2KB:

Replication bandwidth = 10,000 × 2KB = 20 MB/s per direction
                      = 40 MB/s bidirectional
                      = ~3.4 TB/day

At AWS cross-region data transfer pricing (roughly $0.02/GB at scale), that's about $68/day or roughly $2,000/month just for replication bandwidth. For a system doing 100K writes/second, it's $20,000/month in cross-region data transfer alone.

This cost doesn't show up in your compute sizing, but it absolutely shows up in your bill. And it scales linearly with write volume.

graph TD
    A["Write arrives at Region A"] --> B["Write to local DB"]
    B --> C["Enqueue for replication"]
    C --> D["Cross-region network transfer<br/>60-150ms latency<br/>$0.02/GB cost"]
    D --> E["Apply at Region B"]
    E --> F["Conflict check<br/>(if concurrent write exists)"]
    F -->|No conflict| G["Write applied ✅"]
    F -->|Conflict| H["Conflict resolution<br/>(LWW, merge, app-level)"]
    H --> G

This flowchart shows the lifecycle of a write in an active-active multi-region system. A write arrives at Region A, is written to the local database, then enqueued for replication. The cross-region transfer takes 60 to 150 milliseconds and costs approximately $0.02 per gigabyte. Upon arrival at Region B, a conflict check is performed. If no conflict exists, the write is applied. If a concurrent write created a conflict, conflict resolution occurs using last-write-wins, custom merge, or application-level logic before the write is applied.

Replication Lag as a Capacity Variable

In active-passive with asynchronous replication, there's always some lag between the primary and the replica. Under normal conditions, this lag might be seconds. Under load, it can grow to minutes.

Here's why this matters for capacity: if your primary region fails and you failover to the standby, any writes that happened during the replication lag window are lost. This is your Recovery Point Objective (RPO) - the maximum amount of data you're willing to lose during failover.

The capacity planning trade-off is direct: synchronous replication doubles your write latency (because every write must round-trip to the remote region before acknowledging). If your write SLO is 50ms and the cross-region round trip is 80ms, synchronous replication makes your SLO mathematically impossible.

Most production multi-region systems use asynchronous replication and accept the RPO window. The ones that can't (financial transactions, medical records) either use synchronous replication and accept the latency cost, or use clever architectures like conflict-free replicated data types (CRDTs) that can merge divergent state after the fact.

The Production War Story

We had a service at Amazon that started as a single-region deployment in us-east-1. It served US customers. Life was simple. Then the business expanded to Europe, and suddenly we had EU customers experiencing 130ms minimum latency on every request.

The ask was straightforward: deploy to eu-west-1, active-active, geographic routing. The capacity plan looked clean - 60/40 split, 80% headroom in each region for failover, done.

What nobody sized for was the replication conflict rate. Our service had a feature where users could update their preferences, and those preferences affected downstream systems. In single-region, writes to the same user profile were serialized naturally. In active-active with a user who logged in from their US phone and their EU laptop in the same session, we got concurrent writes to the same record from both regions.

The conflict rate was only 0.3% of writes, but resolving each conflict required a read-modify-write cycle that was 10x more expensive than a normal write. That 0.3% conflict rate consumed 3% of our write capacity. And during peak hours with mobile app traffic from transatlantic flights, the conflict rate spiked to 2%, consuming 20% of write capacity in both regions.

The fix was application-level conflict avoidance - routing all writes for a given user to their "home" region regardless of where the read request came from. Reads were served locally for latency, writes were routed to the home region for consistency. It added 80ms to writes for users outside their home region, but eliminated conflicts entirely.

The capacity lesson: in multi-region active-active, your write capacity plan must include conflict resolution overhead. The conflict rate depends on your data model, your traffic patterns, and your user behavior. You can't estimate it from first principles - you have to measure it or build architectural patterns that avoid conflicts entirely.

A Multi-Region Sizing Framework

Here's the framework I use when sizing a multi-region deployment, whether in production or in an interview.

Step 1: Choose Your Architecture

Step 2: Size Each Region

For each region, compute independently:

Region capacity = (region_share × peak_QPS × P99_per_request × safety_margin)
                + failover_headroom
                + replication_overhead

Where:

• region_share is the fraction of traffic this region handles normally
• failover_headroom is the additional capacity needed to absorb the other region's traffic during failover
• replication_overhead is the CPU/memory/bandwidth consumed by cross-region replication

Step 3: Add Replication Bandwidth

Monthly replication cost = write_QPS × avg_write_size × 86400 × 30 × price_per_GB / (1024³)

This number belongs in your capacity plan, not as an afterthought.

Step 4: Define Your Failover SLO

Interview: The multi-region question in interviews: When an interviewer says "Make this system globally available," they're testing whether you can articulate the trade-offs. Don't just say "deploy to two regions." Walk through:

1. Active-active vs active-passive and why you chose one
2. How you'd split traffic (geographic routing, latency-based)
3. What happens to data consistency during replication lag
4. The cost implications - multi-region is expensive, and naming that shows maturity
5. What your failover strategy is and what RTO/RPO you'd target

This is a question where the depth of your answer matters more than the specific architecture you choose.

When Multi-Region Isn't Worth It

I want to end this lesson with something that's sometimes the right answer in interviews and almost always the right answer for early-stage systems: don't go multi-region if you don't need to.

Multi-region adds complexity, cost, and operational burden. It introduces data consistency challenges that don't exist in single-region. It requires cross-region networking, conflict resolution, distributed coordination.

Single-region with N+AZ redundancy gives you:

• 99.99% availability (if you handle AZ failures well)
• Sub-millisecond cross-AZ latency
• Strong consistency (single-leader database, no conflicts)
• Simpler operations
• Lower cost (no cross-region data transfer)

Multi-region is justified when:

• Your users are geographically distributed and latency matters (< 100ms SLO)
• Your availability requirement exceeds what a single region can provide (> 99.99%)
• Regulatory requirements mandate data residency in specific geographic areas
• Your business cannot tolerate a full-region outage of any duration

For most early-stage and mid-stage companies, single-region is the right answer. Know this so you can say it confidently when it's true.

Six Takeaways

1. Active-passive is simpler, active-active is faster. Active-passive gives you a warm standby at lower complexity. Active-active gives global users lower latency but introduces replication conflicts and higher operational burden. Choose based on your actual requirements, not on what sounds more impressive.

2. The cold start trap will ruin your failover. A standby region that's been sitting idle has cold caches, cold connection pools, and stale state. Either keep it warm with synthetic traffic or size it for cold-start performance.

3. Replication bandwidth is a real cost. At 10K writes/second with 2KB average size, you're moving 3.4 TB/day cross-region. This scales linearly with write volume and shows up on your bill whether or not you planned for it.

4. Conflict resolution is a capacity input. In active-active, concurrent writes to the same record from different regions create conflicts that are 5-10x more expensive to resolve than normal writes. Account for the conflict rate in your write capacity.

5. Geographic routing is the right default. Route reads to the nearest region for latency, route writes to the record's home region for consistency. This avoids most conflicts while keeping read latency low.

6. Single-region is often the right answer. Not every system needs multi-region. Saying this in an interview - with the reasoning to back it up - shows more maturity than defaulting to the most complex architecture.

Exercises

Exercise: Starter: You have users in the US (60%), Europe (30%), and Asia (10%). You're deploying to us-east-1 and eu-west-1 with geographic routing. Calculate the traffic split and the compute overhead for active-active with partial failover (80% headroom in each region). What's the total compute cost as a multiple of peak capacity?

Exercise: Intermediate: A messaging service does 50,000 write operations per second (average message size 1KB) in an active-active setup across us-east-1 and eu-west-1. Calculate the monthly cross-region replication bandwidth cost at $0.02/GB. Then estimate the conflict resolution overhead if 0.5% of writes create conflicts that are 8x more expensive to resolve.

Exercise: Advanced: Design a multi-region capacity plan for a financial trading platform where trades must have zero data loss (RPO = 0) and the system must recover within 30 seconds (RTO = 30s). The platform processes 5,000 trades/second, each trade is 4KB, and users are split 70/30 between US and EU. What replication architecture would you use? What's the write latency impact? How would you size each region, and what's the total cost multiplier?

Back to System Design Advanced