AlgoInsight

Free lesson · System Design Crash Course

Throughput vs Latency vs Concurrency

Easy10 min readsystem-designhlddistributed-systems

Lesson 2.2 - Throughput vs Latency vs Concurrency

The three-words lesson.

Here's a conversation I've heard a hundred times. Two engineers are debugging a slow service. One says, "It's too slow - we need to make it faster." The other says, "Yeah, let's add more servers." They add servers. The service is still slow. Both engineers are confused.

The problem in that conversation is that nobody specified what "slow" or "faster" meant. There are at least two completely different ways a service can be slow - high latency, where each individual request takes a long time, and low throughput, where the service can't handle many requests at once. They look similar from the outside (users wait a long time) but they're different problems with different solutions. Adding more servers helps one of them and not the other.

Today's lesson is about clearly distinguishing these. There are actually three related but distinct concepts: latency, throughput, and concurrency. Get the vocabulary right and a lot of confusion evaporates.

By the end of this lesson, you'll be able to read a performance discussion and know exactly which of the three is being talked about - and what to actually do about it.

The Toll Booth Analogy

Let me introduce the three concepts through the analogy that I find most helpful. Imagine a toll booth on a highway.

Latency is how long it takes ONE car to pass through the booth. The driver pulls up, hands over the toll, the gate opens, they drive through. Maybe 5 seconds total. That's the latency of one transaction.

Throughput is how many cars per second can pass through the booth. With one booth and a 5-second latency per car, you can process 1 car every 5 seconds, or 0.2 cars per second. That's the throughput.

Concurrency is how many cars are being processed at the same time. With one booth, only one car is being processed at a time - concurrency of 1.

Now, suppose the highway gets busy. You want to handle more cars per second. What do you do?

Option 1: Make each booth process cars faster. If you can get the per-car time down from 5 seconds to 2 seconds, throughput goes up to 0.5 cars/second. You've reduced latency, which increased throughput.

Option 2: Add more booths. Now you have 10 booths each taking 5 seconds. Throughput is 10 cars / 5 seconds = 2 cars/second. You've added concurrency, which increased throughput. Notice that the latency per car hasn't changed - each individual car still takes 5 seconds.

These two options solve different problems. If your problem is "the booth is too slow per car," you reduce latency. If your problem is "we have a queue of cars stretching back a mile," you add concurrency.

The mistake in the conversation I described at the start is people confusing these. "Add more servers" is the equivalent of adding more toll booths. It doesn't help if each booth is itself too slow. And making each booth faster doesn't help if you only have one booth and the queue is huge.

Precise Definitions

Let me make this more rigorous.

Latency is the time taken for one operation. Measured per request, in milliseconds (or microseconds, or seconds, depending on the system). We covered this in detail in Lesson 2.1 - and you should be thinking in terms of percentiles (P50, P95, P99), not averages.

Throughput is the rate at which operations can be completed, measured in operations per second. For a web service, this is "requests per second" (RPS or QPS - queries per second). For a database, it might be "transactions per second" (TPS). For a network, it's bytes per second.

Concurrency is the number of operations in flight at the same time. If your system can process 8 requests simultaneously, your concurrency is 8. If it processes them one at a time, concurrency is 1.

The three are related but separate.

flowchart LR
    subgraph Low["Single-threaded slow service"]
        L1["Latency: 1000ms per request"]
        L2["Concurrency: 1 at a time"]
        L3["Throughput: 1 req/sec"]
    end

    subgraph Mid["Multi-threaded slow service"]
        M1["Latency: 1000ms per request"]
        M2["Concurrency: 100 at once"]
        M3["Throughput: 100 req/sec"]
    end

    subgraph High["Multi-threaded fast service"]
        H1["Latency: 10ms per request"]
        H2["Concurrency: 100 at once"]
        H3["Throughput: 10,000 req/sec"]
    end

    Low -->|"Add concurrency"| Mid
    Mid -->|"Reduce latency"| High

    style Low fill:#fb7185,stroke:#f43f5e,color:#000
    style Mid fill:#fbbf24,stroke:#f59e0b,color:#000
    style High fill:#34d399,stroke:#10b981,color:#000

A diagram showing three system configurations and how they relate to latency, concurrency, and throughput. The first configuration is a single-threaded slow service  - each request takes 1000 milliseconds, concurrency is 1, throughput is 1 request per second. Add concurrency to make it multi-threaded with 100 in flight  - latency stays at 1000 milliseconds per request, but throughput now reaches 100 requests per second. Now reduce the per-request latency to 10 milliseconds while keeping concurrency at 100  - latency is now 10 milliseconds, throughput becomes 10,000 requests per second. Two separate dials. Adding concurrency multiplies throughput at the same latency. Reducing latency increases throughput further. The combination gets you to high-scale performance.

The diagram captures the central insight: latency and concurrency are two separate dials, both of which affect throughput. You can tune each independently. Confusing them is what makes performance discussions confusing.

Little's Law: The Math

There's a beautiful, simple formula relating all three. It's called Little's Law, named after John Little (the MIT professor who proved it in 1961).

$L = \lambda W$

Where:

In plain English: concurrency equals throughput times latency.

Let's check with our examples. A service with latency 10ms and throughput 1000 req/sec has concurrency:

L = 1000 × 0.010 = 10

So on average, 10 requests are in flight at any moment. That makes sense - each request lasts 10ms, and 1000 arrive per second, so 1000 × 10ms = 10 seconds worth of work distributed across the second.

Why this matters: Little's Law is a constraint that relates the three. If you know any two, you can compute the third. If you want throughput of 10,000 req/sec with latency of 100ms, you need concurrency of 1,000 - which means your system needs to be able to handle 1,000 requests in flight simultaneously. That has implications for everything: thread pool sizes, database connection pools, memory usage, file descriptors.

A real example: when you're sizing a system, you use Little's Law backwards. "We need throughput of 5,000 RPS. Average request takes 50ms. We need concurrency of 5,000 × 0.05 = 250." Now you can size your server fleet - if each server can handle concurrency of 50, you need at least 5 servers (and probably more for redundancy).

You'll see Little's Law come up over and over. Internalize it. Most engineers never explicitly learn it, but senior ones use it constantly, often without naming it.

When You Need Lower Latency vs Higher Throughput

The two needs come from different places. Knowing which one you have is important.

Lower latency matters when: Each user is waiting for the response. Real-time games, video calls, search results, payment confirmations, chat messages. The user is sitting there refreshing or expecting an instant response. Latency directly affects user experience.

Higher throughput matters when: There's a large volume of work to do, but no individual user is necessarily waiting on each piece. Batch processing, analytics pipelines, background jobs, log ingestion, event streaming. You care about total work per hour, not how fast any single operation finishes.

Most user-facing web services need both. The user is waiting (latency matters) and lots of users are using the system simultaneously (throughput matters). Engineers usually optimize one without realizing they're inadvertently degrading the other.

A classic example: batching. If your service does a database write per request, you can dramatically increase throughput by batching writes - collect 100 requests, do one batched write, satisfy all 100. Throughput goes way up. But latency for each individual request goes up too (each waits for its batch to fill before being processed). Whether this is a good trade depends on whether you're optimizing for latency or throughput.

flowchart TB
    subgraph NoBatch["No batching: low latency, lower throughput"]
        N1["1 req/write<br/>5ms latency<br/>200 req/sec"]
    end

    subgraph Batch10["Batch 10: medium of both"]
        B10["10 req/write<br/>50ms latency<br/>1000 req/sec"]
    end

    subgraph Batch100["Batch 100: high throughput, high latency"]
        B100["100 req/write<br/>500ms latency<br/>5000 req/sec"]
    end

    NoBatch -->|"increase batch size"| Batch10
    Batch10 -->|"increase batch size"| Batch100

    style NoBatch fill:#34d399,stroke:#10b981,color:#000
    style Batch10 fill:#fbbf24,stroke:#f59e0b,color:#000
    style Batch100 fill:#fb7185,stroke:#f43f5e,color:#000

A diagram illustrating the trade-off between latency and throughput when batching database writes. Without batching, each request results in one database write that takes 5 milliseconds. Throughput is 200 requests per second. Now batch 10 requests together  - each request now waits up to 50 milliseconds for its batch to fill, but throughput increases to 1000 requests per second. Push further and batch 100 requests together  - latency becomes 500 milliseconds, throughput reaches 5000 requests per second. Latency and throughput are trading against each other. You optimize one by accepting a cost in the other. For user-facing reads, you'd usually want the no-batching option. For background data ingestion, you'd accept the latency cost to get the throughput gains.

This trade-off is everywhere. Buffering, batching, queueing, async processing - these are all techniques that increase throughput at the cost of latency. Whether they're appropriate depends on what matters for your use case.

Where Throughput Hits a Ceiling

A common misconception is that you can keep adding concurrency to scale throughput indefinitely. You can't. Throughput hits a ceiling, set by a bottleneck somewhere in the system.

Possible bottlenecks:

Identifying the bottleneck is a critical performance skill. The classic tool is the USE method (utilization, saturation, errors) by Brendan Gregg - check each resource, find the one that's saturated, that's your bottleneck. Adding more of anything else doesn't help.

When you hit a throughput ceiling, you have to:

  1. Find the bottleneck
  2. Either remove it (scale the bottleneck resource), or
  3. Reduce the work each operation does

Worked Example: Designing for 10,000 RPS

Let me walk through how you'd think about this in a real design conversation.

Requirement: Your service needs to handle 10,000 requests per second. Each request requires a database query with average latency 20ms.

Using Little's Law:

So at any given moment, there are 200 in-flight database queries. Your database needs to handle 200 concurrent queries.

Implications:

What can go wrong:

This is what sizing analysis looks like in practice. Little's Law isn't just an academic curiosity; it's the basic math you use when laying out a system.

Concurrency Is Not Parallelism

One more clarification, because this trips up beginners: concurrency and parallelism are different. Both are about doing multiple things, but in different senses.

Concurrency is about handling multiple things at once. A single CPU core can be concurrent - it can be in the middle of many requests simultaneously, switching between them rapidly. You're not done with any of them, but you're making progress on all.

Parallelism is about executing multiple things at the same instant. A multi-core CPU can be parallel - two cores actually running code at the same physical moment.

You can have concurrency without parallelism (one core handling 1000 requests by time-slicing). You can have parallelism without concurrency (eight cores each running independent computations). And you can have both (eight cores, each time-slicing across many requests).

For throughput, both help. Concurrency helps because work can overlap I/O waits (one request waiting for a database can let another use the CPU). Parallelism helps because actual computation runs faster on multiple cores.

For your purposes early on, the distinction matters less than the principle: you scale throughput by doing more things at once. Whether they're "really" parallel or just "concurrent" is mostly a CPU-architecture detail.

There's a famous Rob Pike talk titled "Concurrency Is Not Parallelism" that goes deeper if you're curious - see references.

Six Takeaways from This Lesson

  1. Latency, throughput, and concurrency are three different things. Latency is per-request time. Throughput is operations per second. Concurrency is operations in flight.

  2. Little's Law connects them: L = λW. Concurrency = throughput × latency. Memorize this - it's the basic math of capacity planning.

  3. Adding concurrency (more servers, threads) increases throughput at the same latency. Reducing latency increases throughput at the same concurrency. These are separate dials.

  4. Latency matters when users are waiting. Throughput matters when there's a volume of work. Most user-facing systems care about both.

  5. Throughput hits a ceiling at the bottleneck resource. CPU, memory, network, database connections, locks, or external dependencies. Find the bottleneck before trying to scale.

  6. Concurrency and parallelism are different. Concurrency is structure (multiple things being handled); parallelism is execution (multiple things happening simultaneously). Both contribute to throughput.

Try It

Exercise: Warmup (10 minutes). Pick a system you're familiar with (a web app, a script, anything). What's its latency for a typical operation? What's its throughput? What's its concurrency at typical load? Don't worry about exact numbers - estimate orders of magnitude.

Exercise: Core (20 minutes). Use Little's Law on a real example. Pick a real or hypothetical scenario: "10,000 users active, each clicking once every 30 seconds, average request latency 100ms." Compute the concurrency. How many database connections would you need? How many app server instances if each handles concurrency of 50?

Exercise: Stretch (45 minutes). Read about the USE method by Brendan Gregg. Pick a system you maintain (or a system at work). Walk through utilization, saturation, and errors for CPU, memory, network, and disk. Where's your bottleneck?

What's Next

Lesson 2.3 turns to availability - what it means for a system to be "up." We'll precisely define the nines (99.9%, 99.99%, etc.) and look at what they actually allow in real downtime. The math is surprising, and so are the practical implications.

Back to System Design Crash Course