Free lesson · System Design Advanced

P99 Sizing vs Average Sizing

Hard10 min readsystem-designadvanced

You've just estimated your system handles 10,000 requests per second with an average latency of 45 milliseconds. Your manager nods. Your interviewer nods. The numbers are clean, the math checks out, and you feel good about it.

Then the system goes to production, and at 2 AM on Black Friday, your P99 latency hits 1,200 milliseconds. Your load balancer starts health-checking your servers as unhealthy. Requests pile up. The cascading failure takes down the entire service for eleven minutes. Your average latency was fine. Your tail latency was a disaster.

This is the lesson about why averages will betray you, and what to use instead.

Why Averages Lie

Here's the thing about averages that most engineers learn the hard way: they hide the exact cases you should be most worried about.

Let me give you a number that should make you uncomfortable. If your service handles 10,000 requests per second and your P99 latency is 1 second, that means 100 requests every single second are experiencing at least 1 second of latency. That's 360,000 slow requests per hour. And your average might still look beautiful at 50 milliseconds because the other 9,900 requests are fast.

The average is the number that makes dashboards green. The P99 is the number that determines whether your users come back tomorrow.

graph LR
    subgraph "10,000 Requests per Second"
        A["9,900 requests<br/>< 50ms ✅"]
        B["99 requests<br/>200-800ms ⚠️"]
        C["1 request<br/>> 1,200ms 🔥"]
    end
    
    D["Average: 52ms<br/>Looks great!"]
    E["P99: 800ms<br/>Problem hiding"]
    F["P99.9: 1,200ms<br/>Users leaving"]
    
    A --> D
    B --> E
    C --> F

This diagram shows how 10,000 requests per second distribute across latency buckets. 9,900 requests complete under 50 milliseconds, making the average look healthy at 52ms. But 99 requests take 200 to 800 milliseconds, pushing the P99 to 800ms. And 1 request per second exceeds 1,200 milliseconds at the P99.9 level. The average hides the problem that the tail latency reveals.

The Percentile Ladder

When we talk about percentile-based sizing, we're talking about a ladder of increasing specificity:

At Amazon, we had an internal saying that I'll paraphrase: your P50 is your system working. Your P99 is your system under stress. Your P99.9 is your system revealing its actual architecture.

That last one is the insight. Tail latency isn't random noise - it's a signal about structural problems. Garbage collection pauses. Lock contention. Network retries. Disk I/O queueing. Cold cache hits. These problems are invisible at the median but dominate the tail.

The Math That Changes How You Size

Let me walk through the arithmetic that makes tail latency terrifying at scale.

Fan-out Amplification

Here's where it gets real. Most modern systems don't serve requests from a single server. A single user request fans out to multiple backend services. And when you fan out, your tail latency gets multiplied.

Say your user request hits 5 backend services in parallel, and each service has a P99 latency of 100 milliseconds. What's the P99 of the overall request?

It is absolutely not 100 milliseconds.

The probability that all 5 services respond within their P99 is:

P(all 5 under P99) = 0.99^5 = 0.951

That means your overall request only has a 95.1% chance of meeting the per-service P99. Your system's effective P99 is now somewhere around the P95 of any individual service. And that's just 5 services. At 20 services:

P(all 20 under P99) = 0.99^20 = 0.818

Only 81.8% of your requests will have all backends under their P99. Your system-level P99 is now closer to the P80 of individual services.

graph TD
    A["User Request"] --> B["API Gateway"]
    B --> C["Auth Service<br/>P99: 30ms"]
    B --> D["User Service<br/>P99: 45ms"]
    B --> E["Product Service<br/>P99: 60ms"]
    B --> F["Pricing Service<br/>P99: 40ms"]
    B --> G["Inventory Service<br/>P99: 50ms"]
    
    C --> H["Response Assembly"]
    D --> H
    E --> H
    F --> H
    G --> H
    
    H --> I["System P99 ≈ P95 of slowest<br/>≈ 120ms, not 60ms"]

This diagram shows a user request fanning out through an API gateway to five backend services: Auth at P99 30ms, User at P99 45ms, Product at P99 60ms, Pricing at P99 40ms, and Inventory at P99 50ms. The response assembly combines all five. Due to fan-out amplification, the system-level P99 is approximately 120 milliseconds, roughly the P95 of the slowest service, not the 60ms P99 of any individual service.

Important: The Fan-Out Rule: For a request that fans out to N services, each with P99 latency L, the effective system P99 is approximately the P(100 - 100/N) of the slowest service. More services = your tail gets worse. This is why microservice architectures need tighter per-service latency targets than monoliths.

The Right Way to Size for Tail Latency

When you're doing capacity planning, the shift is this:

Old approach (average-based):

• Estimate average QPS → compute average CPU/memory per request → add 30% headroom → done

Better approach (percentile-based):

• Estimate QPS at peak (not average - peak hour, peak day, peak season)
• Compute resource usage at P99 load, not average load
• Account for fan-out amplification across dependent services
• Add headroom based on the percentile your SLO targets, not based on average

The headroom calculation changes dramatically. If your SLO is P99 < 200ms and your current P99 under normal load is 150ms, you do not have 50ms of headroom. You have whatever headroom exists between your P99 under peak load with concurrent failure and 200ms. That number is probably negative, which means you're already undersized.

The Production War Story

At AWS, I was part of a team sizing a new storage tier for S3. The initial capacity plan looked clean - we'd modeled average request rates, average object sizes, average I/O latency. The numbers said we needed 400 servers across three AZs.

We launched with 400 servers. Within a week, the P99 write latency was blowing our SLO by 40%. The problem wasn't the average - it was the distribution. About 2% of our objects were 1000x larger than the median object size. Those large-object writes were causing I/O queue depth spikes on specific servers, and those spikes pushed every other request on those servers into the tail.

The fix wasn't more servers. It was separating the capacity plan into two tiers - one sized for the common case (small objects, latency-optimized) and one sized for the tail case (large objects, throughput-optimized). Same total compute, radically different P99 behavior.

The lesson that stuck with me: the shape of your traffic distribution is a capacity planning input. If you don't know the shape, you don't actually have a capacity plan. You have a guess.

How to Think About Percentile-Based Sizing

Let me give you a practical framework you can use in both production and interviews.

Step 1: Profile Your Traffic Distribution

Before you size anything, answer these questions:

• What's your request rate at average, daily peak, and yearly peak?
• What's the ratio between peak and average? (This is your burst factor - for most web services it's 2-5x, for e-commerce it can be 10x+)
• What does the object/payload size distribution look like? Is it bimodal? Long-tailed?
• Are there different request types with different resource profiles? (Reads vs writes, simple queries vs aggregations)

Step 2: Size for Peak, Not Average

Required capacity = Peak QPS × Per-request resource at P99 × Safety margin

The safety margin is not a fixed percentage. It depends on how variable your tail is:

Step 3: Validate with Load Testing at the Tail

This is where most teams skip and most production incidents start. You need to load test not just at average throughput, but specifically at the percentile targets you care about.

The test that matters is this: run your system at peak QPS for 30 minutes and measure P99 latency during the last 10 minutes (after the system has warmed up and garbage collection has kicked in). If the P99 during sustained peak exceeds your target, you're undersized. Period.

Interview: When an interviewer asks "How would you size this system?"

Start with the average estimation - that shows you know the basics. Then immediately pivot to tail behavior: "But the average doesn't tell us the full story. I'd want to understand the traffic distribution. What's the burst factor? What does the P99 look like under peak load? Because for a system like this with fan-out to N services, our system-level P99 will be significantly worse than any individual service's P99."

That pivot is what separates a junior answer from a senior one.

The Percentile Estimation Cheat Sheet

Here are the numbers that experienced engineers carry around in their heads for quick estimation:

Tip: Notice the pattern: for most systems, P99 is roughly 5-10x the average, and P99.9 is roughly 5x the P99. This isn't a law of physics - it's an empirical pattern across many production systems. When your ratios deviate significantly from this, it's a signal that something structural is happening (bimodal traffic, GC pauses, lock contention).

Putting It All Together: A Sizing Walkthrough

Let's size a notification delivery system that sends push notifications to mobile devices. The kind of thing you'd see in a system design interview.

Requirements:

• 500 million active users
• Average 5 notifications per user per day
• P99 delivery latency under 2 seconds
• System must handle 3x burst during breaking news events

Step 1: Base traffic

Average daily notifications = 500M × 5 = 2.5 billion / day
Average QPS = 2.5B / 86,400 ≈ 29,000 QPS

Step 2: Peak traffic

Daily peak (assume 3x average for evening hours): 87,000 QPS
Breaking news burst (3x daily peak): 261,000 QPS

Step 3: Per-request resource profile at P99

Each notification delivery involves:

• Read user preferences from cache (P99: 2ms, P50: 0.3ms)
• Read device tokens from database (P99: 15ms, P50: 3ms)
• Format notification payload (P99: 5ms, P50: 1ms)
• Enqueue to delivery pipeline (P99: 10ms, P50: 2ms)

Sequential P99 total: ~32ms per notification But these can be partially parallelized. Realistic P99: ~20ms

Step 4: CPU sizing

At 261K QPS (burst), 20ms P99 processing time:
Concurrent requests = 261,000 × 0.020 = 5,220 concurrent requests
At ~100 concurrent requests per 8-core machine: 53 machines
With 1.5x safety margin: 80 machines

Step 5: The question the interviewer is waiting for

"What happens if an AZ goes down during a breaking news burst?"

This is where Lesson A1.2 picks up - but the short answer is: if you've sized for 80 machines across 3 AZs and one goes down, you have 53 machines handling 261K QPS. That puts you right at the edge of your headroom with zero safety margin. Which means you need to size for the burst after losing an AZ, not before.

Six Takeaways

1. Averages hide the tail. Your P99 is the real measure of your system's capacity. The average is the story you tell to feel comfortable; the tail is the story production tells you at 2 AM.

2. Fan-out amplifies tail latency. With N services in the request path, your system P99 is far worse than any individual service P99. The formula is simple: more services, worse tail.

3. Size for peak, not average. Use burst factors specific to your domain (2-5x for typical web, 10x+ for event-driven traffic like e-commerce). Multiply by your P99 per-request cost, not the average.

4. Know the shape of your distribution. Bimodal traffic (many small requests + few huge ones) needs a different capacity strategy than uniformly distributed traffic. The shape is an input, not an afterthought.

5. P99/P50 ratios are diagnostic. When the ratio exceeds 10x, something structural is happening - GC, lock contention, I/O queueing, cache misses. The ratio tells you where to look.

6. Safety margins are not flat percentages. A tight-distribution system needs 30% headroom. A heavy-tailed system might need 100%. The margin should reflect your actual tail behavior, not a default number.

Exercises

Exercise: Starter: You have a web API with P50 latency of 10ms and P99 latency of 200ms. The API fans out to 3 backend services. Estimate the system-level P99 latency if each backend has the same distribution.

Exercise: Intermediate: A video streaming service has 100M daily active users, each streaming an average of 2 hours per day. The average bitrate is 5 Mbps, but 10% of users watch 4K at 25 Mbps. Size the CDN edge capacity for both average and P99 bandwidth, accounting for a 4x evening peak.

Exercise: Advanced: Design a capacity planning model for a ride-sharing service where demand has both predictable peaks (rush hours, weekends) and unpredictable bursts (weather events, concerts). How would you differentiate your sizing for the predictable vs unpredictable components? What headroom strategy would you use for each?

Back to System Design Advanced