Free lesson · System Design Crash Course

Latency, Precisely (P50, P95, P99, P999)

Easy10 min readsystem-designhlddistributed-systems

Lesson 2.1 - Latency, Precisely (P50, P95, P99, P999)

The percentile lesson.

There's a story I tell sometimes to make this lesson stick. Imagine you run a small coffee shop. You're proud of the service, so you start tracking how long customers wait. After a week, you compute the average wait time. It's 3 minutes. Great, you think. Three minutes is fast. You're proud.

Then one day you stand in your own shop during a busy morning and you notice something. Most people get served in about 90 seconds. But occasionally - maybe one customer out of every 20 - someone waits 8 or 10 minutes. They're not happy. They tell their friends. Some of them never come back.

Your average - 3 minutes - is technically correct. But it's hiding the experience your slowest customers have. If you only look at the average, you'll never solve the problem. The unhappy customers are invisible in your data.

This is what's happening to almost every web service in the world. They report "average latency" numbers that look fine, while real users are having terrible experiences in the long tail. The fix is to stop looking at averages and start looking at percentiles. That's what this lesson is about.

By the end of this lesson, you'll understand what P50, P95, P99, and P999 mean; why averages are dangerously misleading in distributed systems; and why senior engineers care about tail latency more than they care about the median.

What Latency Is

Quick refresher first. Latency is the time it takes for one operation to complete. That's it. Press play on Netflix - the time between your click and the video starting is latency. Hit "Send" on an email - the time until the email is actually delivered is latency. Make an API call - the time between sending the request and getting the response is latency.

Latency is usually measured in milliseconds (ms) for web services. A "fast" API call might be 50ms. A "slow" one might be 500ms. Anything above a couple of seconds and users start abandoning. For real-time systems (games, video calls, trading), latency requirements are tighter - single-digit milliseconds.

Latency is per-operation. Each request has its own latency. If you send 1,000 requests, you have 1,000 latency measurements. That's where things get interesting.

Why Averages Lie

When you have 1,000 latency measurements, the natural thing to do is compute the average. Add them up, divide by 1,000, get a number. People do this constantly. Dashboards show "avg latency." Performance reports say "average response time was 80ms." It feels like a reasonable summary.

It's almost always misleading.

Here's why. Latency distributions are rarely shaped like the bell curves you learned about in statistics. They're almost always long-tailed - most requests are fast, but some are dramatically slower. The average gets pulled around by the slow ones.

Let me show you with a small example. Suppose you collect 10 latency measurements:

30ms, 45ms, 50ms, 55ms, 60ms, 65ms, 70ms, 80ms, 95ms, 5000ms

Nine of these are between 30ms and 100ms. That last one is 5 seconds - a giant outlier. What's the average?

Sum = 30+45+50+55+60+65+70+80+95+5000 = 5,550ms Average = 5550 / 10 = 555ms

The average says your service is taking 555ms. But that doesn't describe any actual user's experience. Nine of them got responses in under 100ms. One got an awful 5-second wait. The average is wrong about both groups.

Worse: averages get worse when you have more outliers. Add another 5-second outlier and the average jumps to 1,000ms. Add a 30-second one and it explodes. Real production systems have these outliers all the time - slow garbage collection, occasional network hiccups, database lock contention, you name it. Averages get warped by them.

The fix is percentiles.

What Percentiles Actually Mean

A percentile is just a way of describing where a value sits in a sorted list.

Imagine sorting your 10 measurements above from smallest to largest:

30, 45, 50, 55, 60, 65, 70, 80, 95, 5000

Now think about each position. The 50th percentile (P50) is the value where 50% of measurements are at or below it. In our 10 measurements, that's around position 5, so somewhere around 60ms. (For a small list there's some debate about exactly how to compute it - interpolation between adjacent values is standard - but the basic idea holds.)

Similarly:

• P95 = 95% of measurements are ≤ this value. In a list of 10, that's around the 9.5th position, so around 95ms.
• P99 = 99% of measurements are ≤ this value. Around position 9.9 - for this small list, basically the maximum.
• P999 (P99.9) = 99.9% are ≤ this value. Doesn't make much sense with only 10 measurements; you need ≥1,000.

Now here's the key insight. The P50 (also called the median) is what the typical user experiences. Half of users had a faster experience; half had a slower one. The P99 is what your unlucky 1% experiences. The P999 is what your truly unlucky 0.1% experiences.

In our small example:

• P50 ≈ 60ms - that's the typical experience
• P99 ≈ 5000ms - that's the unlucky 1% experience
• Average ≈ 555ms - that's nothing real; it's a mathematical artifact

The average tells you about no one. The percentiles tell you about specific groups of users.

The Shape of Real Latency Distributions

Here's roughly what a real latency distribution looks like. Most requests are clustered around some median value. There's a long tail of slower requests stretching out far to the right:

graph LR
    A["Most requests<br/>(40-100ms)<br/>~95% of traffic"]
    B["Some slow ones<br/>(100-500ms)<br/>~4% of traffic"]
    C["A few really slow<br/>(500ms-5s)<br/>~1% of traffic"]
    D["Occasional disasters<br/>(>5s)<br/>~0.1% of traffic"]

    A --> B --> C --> D

    style A fill:#34d399,stroke:#10b981,color:#000
    style B fill:#fbbf24,stroke:#f59e0b,color:#000
    style C fill:#fb923c,stroke:#ea580c,color:#000
    style D fill:#fb7185,stroke:#f43f5e,color:#000

A diagram showing how latency typically distributes in production systems. About 95% of requests are clustered in the fast region  - 40 to 100 milliseconds. About 4% are somewhat slow  - 100 to 500 milliseconds. About 1% are really slow  - 500 milliseconds to 5 seconds. And about 0.1% are absolute disasters  - more than 5 seconds. This long-tail shape is what real latency looks like. The fast cluster is what the median and P95 capture. The slow tail is what P99 and P999 capture. If you only look at the average, you'll see something in between that doesn't describe any of these groups accurately.

Notice the colors get darker as you move right. That's not just visual. The right tail is where users get angry, abandon transactions, and tell their friends not to use your service. The left cluster is where users are happy. The whole game of performance engineering is about reducing the size of that right tail.

Why the Tail Matters More Than You'd Think

You might think: "OK, 1% of requests are slow. That's only 1%. What's the big deal?"

Here's the catch: 1% sounds small until you realize what it means at scale. Suppose your service handles a million requests per day. 1% of a million is 10,000 requests per day that experience the slow tail. Every single one is a real user who had a bad time. At the scale of Google or Netflix, that 1% is millions of unhappy users every day.

It gets worse. In any system where one user-facing request triggers multiple internal requests, the tail compounds.

Suppose your service makes 10 internal calls (to a database, a cache, a recommendations service, an authentication service, etc.) to serve one user request. Each internal call has 1% chance of being slow (P99 latency or worse). The probability that AT LEAST ONE of those 10 calls is slow is:

$1 - (1 - 0.01)^{10} = 1 - 0.99^{10} \approx 9.6%$

So instead of 1% slow requests, you get nearly 10% slow user-facing requests. The tail latencies of your dependencies have stacked up.

Now scale this up. A Google search query might call 100 internal services. With each at 1% P99, the probability of at least one slow call is:

$1 - 0.99^{100} \approx 63%$

63% of user-facing requests will see at least one slow internal call. This is why Google has been obsessed with tail latency for over a decade. It's why they wrote papers about it (see references). At their scale, the tail is the whole story.

This phenomenon is called "the tail at scale," named after a famous 2013 paper by Jeff Dean and Luiz Barroso. Worth reading if you want to go deeper.

Why P50 Isn't Enough

The natural follow-up: "OK, so I should use the median (P50) instead of the average?"

No. P50 is also not enough on its own. The median tells you the typical experience but says nothing about the tail. A service with P50 = 50ms and P99 = 100ms is great. A service with P50 = 50ms and P99 = 5 seconds is terrible. Both have the same P50.

The standard practice is to track multiple percentiles simultaneously: P50, P95, P99, and P999. Together, they describe the shape of the distribution.

Here's an example dashboard for a typical web service:

Service: User-Auth-API
24-hour latency:
  P50:  45ms   ← half of requests faster, half slower
  P95:  120ms  ← 95% finish within 120ms
  P99:  280ms  ← 99% finish within 280ms
  P999: 1200ms ← 99.9% finish within 1.2s

You can read this whole picture. Most users see fast responses (P50 = 45ms). The unlucky 1% wait nearly 300ms (P99 = 280ms). The truly unlucky 0.1% wait over a second (P999 = 1.2s).

When something goes wrong, percentiles tell you where the problem is. If P50 jumps but P99 doesn't change much, you have a system-wide slowdown. If P99 spikes but P50 is fine, you have intermittent issues affecting some requests but not most. The shape of how percentiles move is itself a diagnostic.

How to Actually Measure Percentiles

There's a subtle problem with percentiles: they're harder to compute correctly than averages.

Computing an average is easy: keep a running sum and a running count. Divide at the end. The math works incrementally and uses constant memory.

Computing exact percentiles is harder. To find the 99th percentile of N measurements, you (conceptually) need to sort all N values and pick the right position. That requires storing all the measurements, which doesn't scale.

In practice, monitoring systems use histograms - they bucket measurements by range (e.g., "0–10ms", "10–20ms", "20–50ms", etc.) and count how many fall in each bucket. From the histogram, percentiles can be estimated. There are sophisticated data structures (HDR Histogram, T-Digest, DDSketch) that give accurate percentile estimates with bounded memory.

The practical implication: when you set up monitoring, use a tool that supports histogram-based percentile tracking. Prometheus has histogram metrics. Datadog has distribution metrics. New Relic has percentiles. These compute percentiles correctly. Don't use a tool that only tracks averages.

There's also a common mistake to watch out for: you cannot average percentiles. If your service has P99 of 100ms in region A and 200ms in region B, the global P99 is NOT (100+200)/2 = 150ms. Combining percentiles requires either the underlying raw data or histograms - not the percentile numbers themselves. A lot of dashboards get this wrong, including some expensive enterprise ones.

A Practical Note on Service-Level Objectives

When you set targets for your service, set them as percentiles, not as averages.

Bad SLO: "Average latency under 200ms" Good SLO: "99% of requests under 200ms" (i.e., P99 ≤ 200ms)

The good SLO is meaningful. The bad SLO can be technically satisfied while users have horrible experiences.

At companies like Google and Amazon, virtually all service SLOs are stated as percentiles. The most common are P95, P99, and sometimes P999 for critical services. "How fast" is always answered as "for what percentage of users."

You'll see this come up again in Lesson 2.3 when we talk about availability. Percentiles aren't just for latency - they're a general way to describe distributions of measurements.

A Concrete Improvement Loop

Here's how senior engineers actually use latency percentiles in their daily work:

1. Set targets per percentile. P50, P95, P99 each have a target.
2. Monitor with histograms. A dashboard shows current values for all four percentiles, over time.
3. Alert on tails. If P99 climbs above the target, an alert fires. Average and P50 are less useful for alerting - they move slower than tails.
4. Diagnose by comparing percentiles. Did P50 move? P99? P999? The pattern tells you what changed.
5. Optimize the tail. Most optimization work targets the long tail, not the median. That's where unhappy users live.

This is a workflow, not just a measurement. Latency percentiles inform daily decisions about what to investigate, what to optimize, and what to ship.

Six Takeaways from This Lesson

1. Averages lie. They get pulled by outliers and describe no real user's experience in long-tailed distributions.

2. Use percentiles instead. P50 (median), P95, P99, P999. Each describes a different slice of users.

3. P50 = typical user. P99 = unlucky 1%. P999 = unlucky 0.1%. These groups have very different experiences in the same system.

4. The tail compounds at scale. A service calling 10 dependencies with 1% slow each gets nearly 10% slow user-facing requests. With 100 dependencies, it's 63%.

5. Track multiple percentiles together. P50 + P95 + P99 describes the distribution's shape better than any single number. The way they move during an incident is itself a diagnostic.

6. Use histogram-based metrics, not averages. And never average percentiles - that math is wrong. Use proper tools (Prometheus, Datadog, etc.) that handle this correctly.

Try It

Exercise: Warmup (10 minutes). For a service you've built or used, write down what you think the P50, P95, P99 latency are. You probably don't know the exact numbers, but think about the orders of magnitude. Are they 10ms? 100ms? 1 second?

Exercise: Core (20 minutes). Pull up the metrics for a service at work. Find the latency dashboard. If it only shows average, that's a problem - note it and try to add P99 tracking. If it shows percentiles, study them over the last week. Are there moments when P99 spiked while P50 stayed flat? What happened then?

Exercise: Stretch (45 minutes). Watch Gil Tene's "How NOT to Measure Latency" talk. It's about an hour but worth it. Take notes on what he says about coordinated omission - a subtle measurement bug that affects most percentile measurements in the wild.

What's Next

Lesson 2.2 clarifies the relationship between latency, throughput, and concurrency. These three are often confused, but they're separate concerns. Once you know the precise distinctions, design conversations get much sharper.

Back to System Design Crash Course