Free lesson · System Design Interviews

Step 7: Deep Dive - Scaling Design

Hard10 min readsystem-designinterview-prep

Your high-level architecture works at moderate scale - a few thousand QPS, a single database, a small Redis cluster. But the interviewer's next question is always: "How does this scale to 10x? 100x?" This deep dive is about showing that you understand where the system breaks under load and how to fix each breaking point without over-engineering from the start.

At Amazon, we call this "designing for the next order of magnitude." You don't build for 100x on day one - you build for 1x with a clear plan for 10x and a rough sketch for 100x. Let me walk through each layer and show where the bottlenecks emerge and how to address them.

Scaling the Read Path

The read path handles redirects - 100K QPS average, 300K peak. This is where scaling matters most because it's 100x the write traffic.

Layer 1: Add a CDN for Global Reach

Before traffic even reaches your servers, a CDN can handle a significant portion of redirect requests. The CDN caches the 302 redirect response for popular URLs.

flowchart LR
    USER["User in Tokyo"] --> CDN_EDGE["CDN Edge Node<br/>(Tokyo POP)"]
    CDN_EDGE -->|"Cache HIT"| REDIRECT_CDN["302 Redirect<br/>Latency: ~5ms"]
    CDN_EDGE -->|"Cache MISS"| ORIGIN["Origin Servers<br/>(US-East)"]
    ORIGIN --> REDIRECT_ORIGIN["302 Redirect<br/>Latency: ~150ms"]
    
    USER2["User in NYC"] --> CDN_EDGE2["CDN Edge Node<br/>(NYC POP)"]
    CDN_EDGE2 -->|"Cache HIT"| REDIRECT_CDN2["302 Redirect<br/>Latency: ~2ms"]

CDN layer for global scaling. A user in Tokyo hits the nearest CDN edge node. If the redirect is cached at the edge, the response returns in about 5 milliseconds without ever reaching the origin servers. If it's a cache miss, the request travels to the origin in the US, adding about 150 milliseconds of latency. For popular URLs, the CDN absorbs the majority of global traffic, reducing origin load by 60 to 80 percent.

CDN caching for 302 redirects requires careful configuration. By default, CDNs don't cache 302 responses because they're "temporary." You need to add explicit cache headers:

HTTP/1.1 302 Found
Location: https://www.example.com/long-url
Cache-Control: public, max-age=300    # Cache for 5 minutes at edge
X-Short-Code: a3Xb9k                 # For cache key identification

The trade-off: CDN-cached redirects bypass your click tracking. If a user in Tokyo hits a CDN edge cache, the click event never reaches your analytics pipeline. Solutions:

Interview: What to say: "I'd add a CDN layer for global reach. Popular URLs get cached at edge nodes - a user in Tokyo gets a redirect in 5ms instead of 150ms. The trade-off is that CDN-cached redirects bypass our click tracking. For analytics accuracy, I'd use a short CDN TTL of 5 minutes and integrate CloudFront access logs with our analytics pipeline. We get 80% of the CDN benefit while retaining most click data."

Layer 2: Horizontal Scaling of Read Servers

Read servers are stateless - they just look up a cache key and return a redirect. Scaling them horizontally is straightforward:

Current: 10 read servers handling 100K QPS (10K per server)
10x scale: 100 read servers handling 1M QPS
100x scale: 1,000 read servers handling 10M QPS

Each read server needs no local state - the cache lookup goes to Redis, and Redis handles the data distribution. Adding servers is a load balancer configuration change.

Auto-scaling policy: Scale based on CPU utilization (target: 60%) or request count per target. During traffic spikes (viral content, marketing campaigns), new servers spin up in 2-3 minutes with container orchestration (ECS or Kubernetes).

Layer 3: Scaling the Redis Cache

A single Redis instance handles ~100K ops/sec. For 300K peak QPS, you need a Redis Cluster:

flowchart TB
    subgraph CLUSTER["Redis Cluster (6 nodes)"]
        subgraph SHARD1["Shard 1"]
            P1["Primary 1<br/>Keys: a-j"]
            R1["Replica 1"]
        end
        subgraph SHARD2["Shard 2"]
            P2["Primary 2<br/>Keys: k-s"]
            R2["Replica 2"]
        end
        subgraph SHARD3["Shard 3"]
            P3["Primary 3<br/>Keys: t-z, 0-9"]
            R3["Replica 3"]
        end
    end
    
    APP["Read API Servers"] -->|"Consistent hashing<br/>routes to correct shard"| CLUSTER

Redis Cluster for scaling the cache layer. The cluster has 3 shards, each with a primary and replica node - 6 nodes total. Keys are distributed across shards using consistent hashing based on the short code. Shard 1 handles keys starting with a through j, Shard 2 handles k through s, and Shard 3 handles t through z and digits. Each shard handles about 100,000 operations per second, giving the cluster a combined capacity of 300,000 operations per second. Replicas provide failover if any primary goes down.

Scaling Redis further:

• Read replicas per shard: Add 2 more replicas per shard, route read traffic to replicas. Triples read capacity to ~900K ops/sec.
• More shards: Add shards to distribute keys further. Each shard added increases capacity by ~100K ops/sec.
• Client-side caching: Read servers maintain a small local LRU cache (100MB) for the hottest ~500K URLs. Reduces Redis traffic by 20-30% for Zipf-distributed access patterns.

Scaling the Write Path

The write path handles URL creation - 1K QPS average, 3K peak. This is lower traffic but has different bottlenecks.

Database Write Scaling

A single PostgreSQL primary handles 5-10K simple INSERTs per second. At 3K peak, we have headroom. But at 10x scale (30K peak), a single primary can't keep up.

Approach 1: Sharding by short_code

flowchart TB
    WRITE_API["Write API"] --> ROUTER["Shard Router<br/>hash(short_code) % N"]
    
    ROUTER -->|"hash = 0"| DB1[("Shard 0<br/>PostgreSQL")]
    ROUTER -->|"hash = 1"| DB2[("Shard 1<br/>PostgreSQL")]
    ROUTER -->|"hash = 2"| DB3[("Shard 2<br/>PostgreSQL")]
    ROUTER -->|"hash = 3"| DB4[("Shard 3<br/>PostgreSQL")]

Database sharding for write scaling. The write API uses a shard router that hashes the short code and routes the INSERT to the correct shard. With 4 shards, each shard handles one-quarter of the write traffic - about 750 QPS average instead of 3,000. This also distributes storage across shards, so each shard holds about 14 terabytes instead of 55 terabytes over 5 years. The redirect path benefits too - each cache miss query hits exactly one shard because we shard by the same key used for lookups.

Why shard by short_code: The redirect path looks up by short_code. Sharding by the same key means every redirect query (cache miss) hits exactly one shard - no scatter-gather. This is the critical optimization for a read-heavy system.

Approach 2: Separate write database from read database (CQRS)

Write path: Write API → PostgreSQL Primary (optimized for INSERTs)
Read path: Read API → Redis Cache → PostgreSQL Read Replicas (optimized for SELECTs)

Replication: Primary → Read Replicas (async, <1 second lag)

This is simpler than sharding and works until the primary can't handle write throughput alone. At 10K write QPS, CQRS with a single primary is sufficient. At 30K+, you need sharding.

Short Code Generator Scaling

The pre-allocated counter range approach already scales horizontally - each write server has its own range and increments independently. Adding more write servers just means allocating more ranges. The range allocation service itself is low-traffic (one request per million URLs) and doesn't become a bottleneck.

Scaling for Traffic Spikes

URL shorteners experience extreme traffic spikes when a short URL goes viral. A single URL might go from 0 clicks/sec to 50,000 clicks/sec in under a minute.

The Viral URL Problem

flowchart TD
    VIRAL["Viral URL shared on Twitter<br/>50,000 clicks/sec for 1 URL"] --> CACHE_CHECK{"Is it in cache?"}
    
    CACHE_CHECK -->|"YES"| FINE["Redis serves 50K/sec<br/>for a single key.<br/>No problem - Redis<br/>handles this easily."]
    
    CACHE_CHECK -->|"NO (expired/evicted)"| STAMPEDE["Cache stampede!<br/>50,000 simultaneous<br/>cache misses"]
    
    STAMPEDE --> COALESCE["Request coalescing:<br/>1 DB query, 49,999 wait"]
    COALESCE --> REPOPULATE["Cache repopulated<br/>in ~10ms.<br/>All 49,999 requests<br/>get cache hit."]

Handling a viral URL spike. When a URL goes viral and receives 50,000 clicks per second, the system behaves differently depending on whether the URL is cached. If it's in cache, Redis handles 50,000 reads per second for a single key without problems. If the URL has been evicted from cache, 50,000 simultaneous cache misses occur. Request coalescing ensures only one database query fires while the other 49,999 requests wait. The cache is repopulated in about 10 milliseconds, after which all waiting requests receive cache hits.

Hot key mitigation: Even with the URL in cache, 50K reads/sec for a single key means one Redis shard handles all that traffic. If one key receives disproportionate traffic, it creates a "hot shard" problem.

Solutions:

• Local caching on read servers: Each read server caches the top 1,000 URLs in local memory (LRU, 100MB). Viral URLs get served from local memory without hitting Redis at all.
• Key replication: Replicate the hot key across multiple Redis shards with a suffix: url:a3Xb9k:0, url:a3Xb9k:1, url:a3Xb9k:2. Read servers pick a random suffix, distributing load across shards.

Tip: Proactive hot-key detection. Monitor Redis key access frequency. When a key exceeds 10K reads/sec, automatically promote it to the local cache on all read servers. This prevents the hot shard problem before it causes latency degradation.

Scaling the Analytics Pipeline

At 100K redirects/sec, the analytics pipeline ingests 100K click events per second. Kafka handles this comfortably with a few partitions. At 10x (1M events/sec), you need more partitions and consumers.

flowchart LR
    subgraph PRODUCERS["Read API Servers (100 instances)"]
        P1["Server 1"]
        P2["Server 2"]
        P3["..."]
    end
    
    subgraph KAFKA["Kafka Cluster"]
        TOPIC["click-events topic<br/>12 partitions"]
    end
    
    subgraph CONSUMERS["Analytics Workers"]
        C1["Worker 1<br/>(partitions 0-3)"]
        C2["Worker 2<br/>(partitions 4-7)"]
        C3["Worker 3<br/>(partitions 8-11)"]
    end
    
    PRODUCERS --> KAFKA
    KAFKA --> CONSUMERS
    CONSUMERS --> DB[("Analytics DB<br/>(batch updates)")]

Scaling the analytics pipeline. One hundred read API servers produce click events into a Kafka topic with 12 partitions. Three analytics worker instances consume from the partitions - each worker handles 4 partitions. Workers batch click events and update the analytics database in bulk rather than one-by-one, achieving much higher throughput. To scale further, add more Kafka partitions and more workers.

Batch updates for efficiency: Instead of incrementing click_count for every click individually, the analytics worker batches clicks by short_code and issues bulk updates:

-- Instead of 1,000 individual UPDATEs per second:
UPDATE url_mappings SET click_count = click_count + 1 WHERE short_code = 'abc123';
UPDATE url_mappings SET click_count = click_count + 1 WHERE short_code = 'xyz789';
-- ... 998 more

-- Batch: one UPDATE every 10 seconds with accumulated counts:
UPDATE url_mappings SET click_count = click_count + 47 WHERE short_code = 'abc123';
UPDATE url_mappings SET click_count = click_count + 12 WHERE short_code = 'xyz789';
-- ... fewer, larger updates

The Scaling Roadmap

Here's how the architecture evolves at each order of magnitude:

Warning: Don't present the 1000x architecture in a 45-minute interview. Start with 1x. When the interviewer asks "how would you scale this?", walk through 10x. If they push further, sketch 100x. Nobody expects you to design for 100M QPS - they want to see that you know where the bottlenecks are and what tools address them.

Multi-Region Architecture (100x Scale)

At 100x scale, you need servers in multiple geographic regions. The question is: does every region have its own database, or do they share one?

flowchart TB
    subgraph US["US-East Region"]
        LB_US["Load Balancer"]
        API_US["Read/Write Servers"]
        CACHE_US[("Redis")]
        DB_US[("PostgreSQL<br/>Primary")]
    end
    
    subgraph EU["EU-West Region"]
        LB_EU["Load Balancer"]
        API_EU["Read/Write Servers"]
        CACHE_EU[("Redis")]
        DB_EU[("PostgreSQL<br/>Replica")]
    end
    
    subgraph ASIA["AP-Southeast Region"]
        LB_ASIA["Load Balancer"]
        API_ASIA["Read/Write Servers"]
        CACHE_ASIA[("Redis")]
        DB_ASIA[("PostgreSQL<br/>Replica")]
    end
    
    DNS["GeoDNS<br/>Routes users to<br/>nearest region"] --> LB_US
    DNS --> LB_EU
    DNS --> LB_ASIA
    
    DB_US -->|"Async replication"| DB_EU
    DB_US -->|"Async replication"| DB_ASIA

Multi-region architecture for 100x scale. Three regions - US East, EU West, and Asia Pacific - each with their own load balancer, API servers, and Redis cache. GeoDNS routes users to the nearest region. The US East region hosts the primary PostgreSQL database. EU and Asia have read replicas with asynchronous replication. Writes always go to the US primary. Reads are served locally from each region's cache and replica. A URL created in the US is available in Europe within 1-2 seconds of replication lag.

Write routing: All writes go to the US-East primary. A user in Tokyo creating a URL experiences ~150ms additional latency for the cross-region write. Reads are served locally from each region's cache and database replica.

Alternative: Active-active with conflict resolution. Each region has its own primary database and accepts writes locally. Conflicts (two regions creating the same short code simultaneously) are resolved by the pre-allocated counter range approach - each region gets its own counter ranges, guaranteeing no conflicts.

Key Takeaways

1. Scale in order of magnitude steps - 1x → 10x → 100x. Don't jump to the most complex architecture. Show the interviewer you can identify when each scaling technique becomes necessary.

2. CDN is the first scaling lever for a read-heavy global system. A CDN absorbs 60-80% of redirect traffic at edge nodes. The trade-off is reduced analytics accuracy, which can be mitigated with edge-side logging.

3. Read servers scale horizontally because they're stateless. Adding more read servers is a load balancer config change. Auto-scale based on CPU or request count.

4. Redis Cluster scales with more shards. Each shard adds ~100K ops/sec capacity. Hot keys are mitigated with local caching on read servers.

5. Database sharding by short_code optimizes the dominant read path. Every redirect cache miss hits exactly one shard. Start with CQRS (read replicas) and move to sharding when write throughput exceeds a single primary.

6. Multi-region is the 100x answer. GeoDNS routes to nearest region. Each region has its own cache and database replica. Writes go to the primary region. Pre-allocated counter ranges prevent cross-region conflicts.

Exercise: Exercise 1: Calculate the cost of the 10x architecture. Estimate monthly AWS costs for: 100 c5.large read servers, 9 Redis nodes (r6g.xlarge), 4 PostgreSQL shards (db.r6g.2xlarge), 1 Application Load Balancer, CloudFront CDN serving 1M requests/sec. Is this cost-effective for a URL shortener business?

Exercise 2: The interviewer asks: "A country-level firewall blocks your CDN. Users in that country can't access short URLs. How do you handle this?" Design a fallback for regions where your CDN doesn't work. (Hint: direct DNS routing to the nearest origin region, bypassing CDN.)

Exercise 3: Design the auto-scaling policy for read servers. What metric do you scale on - CPU, memory, request count, or latency? What's the scale-up threshold and scale-down threshold? How do you prevent thrashing (rapid scale-up/scale-down cycles)?

Back to System Design Interviews