Free lesson · System Design Interviews

Handling Traffic Spikes - Building Systems That Survive the Unexpected

Medium10 min readsystem-designinterview-prep

Here's a story from my time at Stripe. Black Friday 2019. We'd planned for 3x normal traffic. We got 7x. Not because our estimates were wrong - our merchants' marketing campaigns were more successful than they expected. Within 12 minutes, the payment processing latency went from 200ms to 8 seconds. Checkout pages started timing out. Revenue was evaporating for our merchants at $40,000 per minute.

We survived because of a pattern we'd built specifically for this scenario: backpressure propagation with graceful degradation. The system didn't try to process everything at full quality. It shed load intelligently, prioritizing payment completion over analytics, receipts, and webhooks. Those secondary operations queued up and drained over the next 6 hours. Every payment went through. Nothing was lost.

That experience is why I now ask every senior candidate: "Your system is designed for 10,000 requests per second. At 2 AM, it suddenly receives 100,000 requests per second. What happens?" The answer reveals whether they've ever operated a system under real pressure.

Why Spikes Are Different From Steady Growth

Steady traffic growth gives you time. You see the metrics trending up over weeks, you provision more capacity, you're fine. Spikes don't give you time.

Your architecture needs strategies for all three. Predictable spikes use pre-scaling. Semi-predictable spikes use auto-scaling. Unpredictable spikes use backpressure and load shedding - because there's no time to add capacity.

Strategy 1: Rate Limiting - The Front Door Bouncer

Rate limiting is the first line of defense. It rejects excess traffic before it reaches your application servers, preventing cascading overload.

Rate Limiting Algorithms

flowchart LR
    subgraph ALGORITHMS["Rate Limiting Algorithms"]
        direction TB
        TB["Token Bucket<br/>Allows bursts up to bucket size<br/>Refills at steady rate"]
        SW["Sliding Window<br/>Counts requests in rolling window<br/>Smooth, no burst edges"]
        FW["Fixed Window<br/>Counts per time window<br/>Simple but boundary spikes"]
        LC["Leaky Bucket<br/>Processes at fixed rate<br/>Queues excess, drops overflow"]
    end

Four rate limiting algorithms. Token bucket allows bursts up to the bucket capacity and refills at a steady rate. Sliding window counts requests in a rolling time window for smooth limiting. Fixed window counts per fixed time period but can allow burst at window boundaries. Leaky bucket processes at a fixed rate, queuing excess and dropping overflow.

Token Bucket - The Interview Default

The token bucket is the most commonly discussed algorithm and the best default answer:

• The bucket starts with N tokens (the burst capacity)
• Each request consumes one token
• Tokens refill at a steady rate (the sustained rate)
• When the bucket is empty, requests are rejected

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate          # tokens per second
        self.capacity = capacity  # max burst size
        self.tokens = capacity
        self.last_refill = time.time()
    
    def allow(self):
        now = time.time()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.last_refill = now
        
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

Multi-Level Rate Limiting

Don't rate limit at just one layer:

Interview: What to say: "I'd implement rate limiting at three layers. At the edge, Cloudflare rate limits by IP - 10K requests per second per IP, which blocks DDoS and scrapers. At the API gateway, rate limiting by API key enforces plan limits and prevents any single client from overwhelming the system. At the application layer, per-user per-endpoint limits prevent accidental tight loops in client code. The edge limit protects the infrastructure, the gateway limit protects the business model, and the application limit protects the user experience."

Strategy 2: Load Shedding - Dropping Low-Priority Work

When rate limiting isn't enough - when even legitimate traffic exceeds capacity - load shedding drops lower-priority work to protect higher-priority work.

Priority Classification

flowchart TD
    REQUEST["Incoming Request"] --> CLASSIFY{"Priority<br/>Classification"}
    
    CLASSIFY -->|"P0: Critical"| P0["Payment Processing<br/>Login/Auth<br/>Order Submission<br/>→ Always process"]
    
    CLASSIFY -->|"P1: Important"| P1["Feed Loading<br/>Search Results<br/>Product Pages<br/>→ Process if capacity allows"]
    
    CLASSIFY -->|"P2: Deferrable"| P2["Analytics Events<br/>Recommendation Refresh<br/>Email Notifications<br/>→ Queue for later"]
    
    CLASSIFY -->|"P3: Droppable"| P3["Prefetch/Warmup<br/>Non-critical Webhooks<br/>Background Sync<br/>→ Drop during overload"]

Load shedding priority classification. P0 critical requests like payments, auth, and orders are always processed. P1 important requests like feed loading and search are processed if capacity allows. P2 deferrable requests like analytics and notifications are queued for later processing. P3 droppable requests like prefetch and background sync are dropped entirely during overload.

Implementation: The Circuit-Based Approach

class LoadShedder:
    def __init__(self, capacity_threshold=0.85):
        self.threshold = capacity_threshold
    
    def should_process(self, request):
        current_load = self.get_system_load()  # CPU, queue depth, latency
        
        if request.priority == "P0":
            return True  # Always process critical requests
        
        if current_load < 0.70:
            return True  # Under 70% capacity - process everything
        
        if current_load < 0.85:
            return request.priority in ("P0", "P1")  # Shed P2 and P3
        
        if current_load < 0.95:
            return request.priority == "P0"  # Only critical requests
        
        # Over 95% - even some P0 might get queued
        return request.priority == "P0" and random.random() < 0.5

Warning: Load shedding must return fast. A shed request should return 503 Service Unavailable in under 10ms - not sit in a queue. The whole point is to free up resources. If shed requests consume processing time before being rejected, shedding doesn't help.

Strategy 3: Auto-Scaling - Adding Capacity Dynamically

For spikes that last longer than a few minutes, the system should automatically add compute capacity.

Scaling Signals

Scaling Speed

The critical question is how fast new capacity comes online:

flowchart LR
    SPIKE["Traffic Spike<br/>Detected"] --> IMMEDIATE["Immediate (0-30s)<br/>→ Rate limit<br/>→ Load shed<br/>→ Use standby instances"]
    IMMEDIATE --> SHORT["Short-term (30s-5min)<br/>→ Scale containers<br/>→ Scale stateless services"]
    SHORT --> MEDIUM["Medium-term (5-60min)<br/>→ Scale VMs<br/>→ Add cache capacity<br/>→ Add read replicas"]

Three-phase spike response timeline. In the first 30 seconds, use immediate responses: rate limiting, load shedding, and activating standby instances. From 30 seconds to 5 minutes, scale containers and stateless services. From 5 to 60 minutes, scale VMs, add cache capacity, and spin up read replicas. Each phase buys time for the next.

Tip: In interviews, mention the time dimension: "Auto-scaling takes 60-90 seconds to bring new containers online. For the first 60 seconds of a spike, we rely on rate limiting and load shedding to protect the system. Once new instances are healthy and taking traffic, the shedding eases off. This layered response means we handle the first minute with defensive measures and subsequent minutes with increased capacity."

Strategy 4: Backpressure - Slowing Down Instead of Failing

Backpressure is the principle that when a downstream service is overwhelmed, it should signal upstream to slow down rather than accepting requests it can't process.

Without Backpressure

sequenceDiagram
    participant C as Client
    participant API as API Server
    participant Q as Queue
    participant W as Worker
    participant DB as Database
    
    Note over C,DB: Without backpressure - cascading failure
    C->>API: 10,000 req/sec
    API->>Q: 10,000 msg/sec
    Q->>W: Workers pull as fast as they can
    W->>DB: 10,000 writes/sec
    Note over DB: DB handles 5,000/sec max<br/>Connection pool exhausted
    DB--xW: Connection timeout
    W--xQ: Messages pile up (queue grows unbounded)
    Q--xAPI: Queue memory exhausted → crash
    API--xC: 500 Internal Server Error

Cascading failure without backpressure. The client sends 10,000 requests per second. The API server pushes everything to the queue. Workers consume as fast as possible and slam the database with 10,000 writes per second. The database can only handle 5,000 per second, so connections time out. Messages pile up in the queue until memory is exhausted and the queue crashes. The API server returns 500 errors. Everything fails.

With Backpressure

sequenceDiagram
    participant C as Client
    participant API as API Server
    participant Q as Queue
    participant W as Worker
    participant DB as Database
    
    Note over C,DB: With backpressure - controlled degradation
    C->>API: 10,000 req/sec
    API->>Q: Check queue depth
    Note over Q: Queue depth > threshold
    Q-->>API: Backpressure signal
    API-->>C: 429 Too Many Requests (for excess traffic)
    API->>Q: 5,000 msg/sec (admitted traffic)
    Q->>W: Workers pull at sustainable rate
    W->>DB: 5,000 writes/sec (within capacity)
    Note over DB: DB operates normally

Controlled degradation with backpressure. The client sends 10,000 requests per second. The API server checks the queue depth, finds it above threshold, and returns 429 responses for excess traffic while admitting only 5,000 messages per second to the queue. Workers consume at a sustainable rate. The database operates within its capacity. The system is slower for some clients but never crashes.

Implementing Backpressure

Backpressure signals include:

• Queue depth thresholds: If the queue has more than N messages, stop accepting new ones
• Response time thresholds: If downstream latency exceeds M milliseconds, slow the admission rate
• Connection pool saturation: If X% of database connections are in use, reject new requests
• HTTP 429 (Too Many Requests): The standard response code with a Retry-After header

Interview: What to say: "I'd implement backpressure at the queue layer. The API server checks the Kafka consumer lag before accepting new writes. If the lag exceeds 100,000 messages - meaning workers are more than 10 seconds behind - the API server starts returning 429 with a Retry-After header. This prevents the queue from growing unbounded and gives the workers time to catch up. The client retries with exponential backoff. The system degrades gracefully instead of cascading into failure."

Strategy 5: Graceful Degradation - The Feature Toggle Approach

When the system is under extreme load, turn off non-essential features to free up resources for the essential ones.

Feature Priority Map

flowchart TB
    NORMAL["Normal Load<br/>All Features On"] --> ELEVATED["Elevated Load (70-85%)<br/>Disable: personalized recommendations<br/>Serve: cached/generic recommendations"]
    ELEVATED --> HIGH["High Load (85-95%)<br/>Disable: search suggestions, analytics<br/>Serve: simplified search, skip tracking"]
    HIGH --> CRITICAL["Critical Load (95%+)<br/>Disable: images, comments, likes<br/>Serve: text-only core functionality"]
    
    style NORMAL fill:#4CAF50,color:#fff
    style ELEVATED fill:#FFC107,color:#000
    style HIGH fill:#FF9800,color:#fff
    style CRITICAL fill:#f44336,color:#fff

Graceful degradation across four load levels. At normal load, all features are enabled. At elevated load between 70 and 85 percent, personalized recommendations are replaced with cached generic ones. At high load between 85 and 95 percent, search suggestions and analytics tracking are disabled. At critical load above 95 percent, images, comments, and likes are disabled, serving only text-based core functionality.

Real-world example from Amazon: During peak Prime Day traffic, Amazon disables or simplifies features progressively. The product page might lose the "customers also bought" recommendations section, the review section might show fewer reviews, and the "compare with similar items" widget might disappear entirely. The core function - seeing the product, seeing the price, adding to cart, and checking out - is always protected.

Tip: Design your system with feature flags that can be toggled remotely via a configuration service (like LaunchDarkly or a custom feature flag system backed by etcd). During a spike, an on-call engineer - or an automated system - toggles non-essential features off. This is faster than any code deployment and can be reversed instantly.

Putting It All Together - The Spike Response Playbook

flowchart TD
    SPIKE["🚨 Traffic Spike Detected"] --> T1["0-10 seconds"]
    T1 --> RATE["Rate limiting activates<br/>Edge blocks abusive traffic"]
    T1 --> SHED["Load shedding kicks in<br/>P2/P3 requests dropped"]
    
    SPIKE --> T2["10-60 seconds"]
    T2 --> DEGRADE["Feature toggles fire<br/>Non-essential features disabled"]
    T2 --> STANDBY["Standby instances activated"]
    
    SPIKE --> T3["1-5 minutes"]
    T3 --> SCALE["Auto-scaling provisions<br/>new containers"]
    T3 --> CACHE["Cache warming for<br/>hot content"]
    
    SPIKE --> T4["5+ minutes"]
    T4 --> RESTORE["Features gradually restored<br/>as capacity stabilizes"]
    T4 --> REVIEW["On-call reviews<br/>spike cause and duration"]

The spike response playbook in four phases. In the first 10 seconds, rate limiting and load shedding activate automatically. From 10 to 60 seconds, feature toggles disable non-essential features and standby instances come online. From 1 to 5 minutes, auto-scaling provisions new containers and caches warm for hot content. After 5 minutes, features are gradually restored as capacity stabilizes and the on-call team reviews the cause.

Key Takeaways

1. Rate limiting is your first defense. Implement at multiple layers - edge, gateway, application - each protecting against different threat profiles. Token bucket is the default algorithm for interviews.

2. Load shedding preserves critical functionality. Classify requests by priority and drop low-priority work first. P0 requests (payments, auth) are always processed. P3 requests (prefetch, background sync) are dropped during overload.

3. Auto-scaling takes time - bridge the gap. Container scaling takes 60-90 seconds. VM scaling takes 3-5 minutes. Use rate limiting and load shedding during the first minutes while new capacity spins up.

4. Backpressure prevents cascading failures. When downstream services are overwhelmed, signal upstream to slow down. The alternative - accepting requests you can't process - leads to queue exhaustion, timeouts, and system-wide failure.

5. Graceful degradation trades features for stability. Design with feature flags that can disable non-essential features at runtime. During a spike, the system serves a simpler but functional experience rather than failing entirely.

6. Layer your defenses by time horizon. Seconds: rate limiting and load shedding. Minutes: auto-scaling and feature degradation. Hours: manual capacity planning and traffic rerouting. Each layer buys time for the next.

Exercise: Exercise 1: You operate a ticket-selling platform. A major concert goes on sale at noon and you expect 500,000 users to attempt purchases simultaneously for 10,000 available tickets. Design the traffic spike strategy. How do you handle the initial thundering herd? How do you prevent overselling? What does the user experience look like for the 490,000 people who won't get tickets?

Exercise 2: Your API serves both free-tier and paid-tier customers. During a traffic spike, you want to protect paid-tier customer experience at the expense of free-tier. Design a multi-tier rate limiting and load shedding system that gives paid customers priority access during overload while still serving free-tier customers when capacity allows.

Exercise 3: Design an auto-scaling policy for a service that has predictable daily traffic patterns (2x during business hours) and unpredictable spikes (10x for 5-15 minutes). The service runs on Kubernetes. How do you configure the Horizontal Pod Autoscaler? What metric do you scale on? How do you handle the cold-start problem for new pods? What's the cost difference between reactive scaling and keeping warm standby pods?

Back to System Design Interviews