Free lesson · System Design Fundamentals

Latency vs Throughput vs Bandwidth

Easy9 min readsystem-designhlddistributed-systems

Lesson 2.5 - Latency vs Throughput vs Bandwidth

The fifth lesson of Module 2.

Three numbers describe how a system performs - and engineers confuse them constantly. A team says "our system is slow" - do they mean each request takes too long (latency), the system can't handle enough requests per second (throughput), or the network pipe is saturated (bandwidth)? Each has different causes, different symptoms, and different fixes. Conflating them leads to optimizing the wrong thing - tuning query latency when the real bottleneck is connection pool throughput, or adding network bandwidth when the problem is slow database queries.

The Three Numbers, Precisely

Think of a highway between two cities:

• Latency = how long it takes ONE car to drive from City A to City B. Travel time per trip. Measured in milliseconds for network requests.
• Throughput = how many cars pass the toll booth at City B per hour. Flow rate. Measured in requests per second (req/s) or transactions per second (TPS).
• Bandwidth = how many lanes the highway has - the maximum possible flow. The pipe's capacity. Measured in megabits per second (Mbps) or gigabits per second (Gbps).

A highway can have low latency (short distance, high speed limit) but low throughput (one lane, few cars). It can have high bandwidth (10 lanes) but high latency (construction delays, toll booth queues). They're independent dimensions - improving one doesn't automatically improve the others.

flowchart TD
    subgraph Latency["Latency"]
        L["How long does ONE\nrequest take?\nMeasured in ms\n\nUser-facing: 'how fast\ndoes this feel?'"]
    end
    
    subgraph Throughput["Throughput"]
        T["How many requests\nper second?\nMeasured in req/s\n\nCapacity: 'how much\ncan it handle?'"]
    end
    
    subgraph Bandwidth["Bandwidth"]
        B["Maximum data the\nnetwork can carry\nMeasured in Gbps\n\nPipe size: 'how fat\nis the connection?'"]
    end

    style Latency fill:#fbbf24,stroke:#f59e0b,color:#000
    style Throughput fill:#34d399,stroke:#10b981,color:#000
    style Bandwidth fill:#a78bfa,stroke:#8b5cf6,color:#000

Three boxes showing the three performance dimensions. Latency: how long does one request take, measured in milliseconds. This is what users feel - 50ms feels instant, 500ms feels sluggish, 3 seconds feels broken. Throughput: how many requests per second can the system handle, measured in requests per second or transactions per second. This is the system's capacity - can it handle the traffic? Bandwidth: the maximum data the network connection can carry, measured in megabits or gigabits per second. This is the pipe's physical limit. The key relationships: latency and throughput are often inversely correlated under load - as you push throughput toward the maximum, latency increases because requests queue. Bandwidth is a ceiling on throughput for data-heavy workloads - if each response is 1 MB and bandwidth is 1 Gbps (125 MB/s), maximum throughput is 125 responses per second regardless of how fast the server processes them. But for typical API workloads with small JSON payloads, bandwidth is rarely the bottleneck - latency and throughput are.

Latency in Depth

Latency is what users feel. The time from "I clicked the button" to "I see the result." In system design, we measure it as the time from the server receiving a request to sending the response - but end-to-end latency also includes network round-trip time (the client-to-server-and-back trip).

Why percentiles matter more than averages. The average (mean) latency is the most misleading metric in monitoring. Consider this scenario:

Average: (990×50 + 9×500 + 1×8000) / 1000 = 62.5ms. Looks fine on a dashboard.

p50 (median): 50ms. Half of users experience ≤50ms. Good.

p95: 50ms. 95% of users experience ≤50ms. Still looks fine.

p99: 500ms. 1% of users wait half a second. Noticeable but maybe acceptable.

p99.9: 8,000ms. One in a thousand users waits 8 seconds. At 100,000 requests per day, that's 100 users per day with a terrible experience.

The average (62.5ms) hides a problem that affects 100 users daily. Always measure and report p50, p95, p99, and p99.9. Dashboard on p99 at minimum. Alert on p99 or p99.9. Never use the average for latency monitoring - it actively deceives.

Jeff Dean's latency numbers (approximate, memorize the orders of magnitude):

The critical insight: memory (100 ns) is 5,000x faster than same-datacenter network (500 μs). SSD (16 μs) is 10,000x faster than cross-continent network (150 ms). This is why caching exists (M11-M12) - keeping data in memory avoids the network round trip. This is why CDNs exist (M11) - serving content from a nearby edge server avoids the cross-continent round trip. Every major performance optimization in this curriculum is ultimately about avoiding a slow layer by caching at a faster layer.

Throughput in Depth

Throughput is the system's capacity - how many operations per unit of time. A database handles 50,000 queries per second. An API serves 10,000 requests per second. A Kafka cluster ingests 1 million messages per second.

Throughput and latency are inversely correlated under load. This is the most important relationship between the two numbers:

flowchart LR
    subgraph Low["Low utilization (30%)"]
        LL["Latency: 20ms\nThroughput: 300 req/s\nPlenty of headroom"]
    end
    
    subgraph Med["Medium utilization (70%)"]
        ML["Latency: 50ms\nThroughput: 700 req/s\nQueuing starts"]
    end
    
    subgraph High["High utilization (95%)"]
        HL["Latency: 500ms\nThroughput: 950 req/s\nHeavy queuing"]
    end
    
    subgraph Over["Overloaded (>100%)"]
        OL["Latency: timeout\nThroughput: declining\nRequests dropping"]
    end
    
    Low --> Med --> High --> Over

    style Low fill:#34d399,stroke:#10b981,color:#000
    style Med fill:#fbbf24,stroke:#f59e0b,color:#000
    style High fill:#fb7185,stroke:#f43f5e,color:#000
    style Over fill:#7f1d1d,stroke:#991b1b,color:#fff

A progression showing how latency degrades as throughput approaches the system's maximum capacity. At low utilization (30 percent of capacity), the system handles 300 requests per second with 20ms latency per request - plenty of headroom, no queuing, every request is processed immediately. At medium utilization (70 percent), throughput is 700 requests per second but latency has climbed to 50ms - requests occasionally wait briefly for resources (database connections, CPU time, thread pool threads). At high utilization (95 percent), throughput is 950 requests per second but latency has spiked to 500ms - requests are queuing significantly, waiting for busy resources. The system is technically handling the load but users are experiencing degraded performance. At overload (beyond 100 percent capacity), both throughput and latency collapse - the system can't process requests faster than they arrive, queues grow unboundedly, latency climbs to timeout thresholds, and requests start being dropped or erroring. This curve is why rate limiting (M42) and backpressure (M42) exist - they prevent the system from entering the overload zone by rejecting excess traffic before it degrades the experience for everyone.

At low utilization, each request gets immediate access to resources (CPU, database connections, threads) - latency is low. As throughput climbs toward the maximum, resources become contended - requests wait in queues for a database connection, for a thread, for CPU time. Latency increases. Past the saturation point, latency spikes dramatically and throughput actually DECREASES - the system spends more time managing queues than processing requests. This is the throughput-latency curve, and it governs the behavior of every system under load.

This is why capacity planning matters. If your system handles 1,000 req/s at 50ms latency, running it at 950 req/s might push latency to 500ms - a 10x degradation. The safe operating range is typically 60-70% of maximum throughput. Rate limiting (M42) and auto-scaling (M39) keep the system within this range.

Bandwidth in Depth

Bandwidth is the pipe's maximum capacity - the most data that can flow through the network connection per second. A 1 Gbps network link carries ~125 MB per second. A 10 Gbps link carries ~1.25 GB per second.

When bandwidth matters: for typical API workloads (small JSON payloads, 1-10 KB per response), bandwidth is almost never the bottleneck. A 1 Gbps link can carry 125,000 responses of 1 KB each per second - far more than most APIs serve. Bandwidth becomes the constraint for: video streaming (megabytes per second per user), large file transfers (S3 multipart uploads - M24), database replication across regions (M28), ML model distribution (gigabytes of model weights), and backup/restore operations.

The modern reality: within a cloud data center, bandwidth between instances is typically 5-25 Gbps - rarely a bottleneck. Cross-region bandwidth is lower and more expensive - transferring data between AWS regions costs $0.02-0.09/GB, which adds up at terabyte scale. Cross-internet bandwidth (to end users) varies by client connection - a user on fiber has 100+ Mbps; a user on cellular might have 5 Mbps. CDNs (M11) solve this by caching content at edge locations near users, eliminating the long-distance bandwidth constraint.

The Diagnostic Pattern: When Each Is the Problem

The most common misdiagnosis: "our API is slow" → "optimize the database queries." But if the queries are fast at low traffic and slow at high traffic, the problem isn't the queries - it's the throughput bottleneck (exhausted connection pool, full thread pool, saturated CPU). The fix isn't faster queries; it's more capacity or fewer concurrent requests.

A Brandon Story: The Latency That Was Really Throughput

A team I'd advised had an API with a comfortable 50ms p99 latency. Users were happy. Then a marketing campaign drove 5x traffic - from 200 req/s to 1,000 req/s. Latency jumped to 800ms. The team's immediate reaction: "the database queries must be slow - let's optimize them."

But the queries hadn't changed. I asked them to check the database connection pool. They used a pool of 20 connections shared across all API server instances. At 200 req/s, each request got a connection immediately. At 1,000 req/s, requests queued for connections - the pool was saturated. The 750ms of additional latency was wait time in the connection queue, not query execution time.

The fix: increase the connection pool from 20 to 100 connections (and verify the database could handle 100 concurrent connections). Latency at 1,000 req/s dropped back to 60ms. No query optimization needed.

The deeper lesson: they also added monitoring for connection pool metrics - pool utilization, wait time, and queue depth. These metrics (the USE method from M43 - utilization, saturation, errors) would have identified the bottleneck in minutes instead of hours. "High latency under load" is almost always a throughput bottleneck - saturated connection pools, exhausted thread pools, or CPU contention - not a slow individual operation.

Lessons:

• High latency UNDER LOAD is usually a throughput problem, not a latency problem. Diagnose before optimizing.
• Database connection pools, thread pools, and CPU are the three most common throughput bottlenecks.
• Monitor resource utilization and saturation (M43 - USE method), not just request latency.
• Increasing the connection pool from 20 to 100 was a 5-minute configuration change that fixed an 800ms latency problem. No code change required.

Six Takeaways from This Lesson

1. Latency = time for one request. Throughput = requests per second. Bandwidth = maximum data per second. Different dimensions, different fixes.

2. Use percentiles (p50, p95, p99) for latency. Averages hide tail latency that affects real users.

3. Latency and throughput are inversely correlated under load. As throughput approaches max, latency spikes. Operate at 60-70% of capacity.

4. Memory is 5,000x faster than network. SSD is 10,000x faster than cross-continent network. This is why caching and CDNs exist.

5. Bandwidth is rarely the bottleneck for API workloads. It matters for video, large files, and cross-region replication.

6. "High latency under load" is usually a throughput problem - saturated connection pools, thread pools, or CPU. Diagnose with resource utilization metrics, not query profiling.

Try It

Exercise: Warmup (15 minutes). Your API has a p50 of 30ms and a p99 of 2,000ms. At 10,000 requests per minute, how many users per minute experience >2-second latency? Is this acceptable for a checkout page? For a news feed?

Exercise: Core (25 minutes). Your system handles 500 req/s at 50ms latency. Traffic doubles to 1,000 req/s and latency jumps to 400ms. List three possible throughput bottlenecks (connection pool, thread pool, CPU, disk I/O). For each, describe: how would you diagnose it? What metric would you check? What's the fix?

Exercise: Stretch (45 minutes). Read Jeff Dean's "Achieving Rapid Response Times in Large Online Services" (link below). Note: how tail latency (p99, p99.9) compounds in distributed systems - a request that fans out to 100 backend servers has a p99 that's much worse than any individual server's p99.

What's Next

Lesson 2.6 covers consistent hashing - the technique that lets you distribute data across multiple machines without catastrophic reshuffling when machines are added or removed.

Back to System Design Fundamentals