Free lesson · System Design Crash Course

The Latency Numbers to Memorize

Easy10 min readsystem-designhlddistributed-systems

Lesson 3.1 - The Latency Numbers to Memorize

The first practical lesson.

Picture this. You're in a system design interview. The interviewer asks: "Your service needs to look up user profiles for every request. You could store them in memory on the server, in a Redis cache, or in a Postgres database. What would you pick?"

If you don't know the latency numbers, this question feels like a guess. "Um, Redis is fast? Postgres has a lot of features?" Hand-wavy.

If you DO know the latency numbers, the answer comes immediately. Memory access is around 100 nanoseconds. A round trip to Redis on the local network is around 500 microseconds. A Postgres query is around 1–10 milliseconds. Each one is ~1000x slower than the previous. So the answer depends on whether the data fits in memory (best), whether we can tolerate a few hundred microseconds (cache is fine), or whether we need queryable / persistent data (database).

That's what these numbers give you. They turn vibes into reasoning. They let you compare alternatives by their fundamental performance characteristics, not by gut feel. They're the most valuable memorization in this entire course.

Let me introduce them.

The Jeff Dean Story

In the early 2010s, a Google engineer named Jeff Dean - who's a legend in distributed systems; he co-built MapReduce, BigTable, Spanner, TensorFlow, and a lot of what Google runs on - gave a talk where he listed "numbers every engineer should know." A latency cheat sheet. Things like: L1 cache reference is 0.5 ns. Sending 1 KB over a 1 Gbps network is 10 microseconds. A disk seek is 10 milliseconds.

The list took on a life of its own. It got reposted everywhere. Engineers printed it out and put it on their walls. Whole books include some version of it. The reason it stuck wasn't because Jeff Dean is famous - it was because the numbers themselves are genuinely useful. Once you internalize the relative speeds of memory vs cache vs disk vs network, you can reason about almost any performance question.

The original list was from around 2012. Hardware has evolved since then - SSDs got way faster, network speeds in datacenters jumped from 1 Gbps to 25–100 Gbps, memory got a bit quicker. But the shape of the numbers - the orders of magnitude between them - hasn't changed much. Memory is still ~100x faster than the local network. Local network is still ~1000x faster than crossing the continent. The speed of light hasn't changed (and won't).

Below is the list, updated for 2026 hardware. You should aim to internalize these. Not memorize-for-a-test internalize - but recognize the orders of magnitude and recall them in conversation. We'll go through how to use them in a moment.

The Numbers

Here's the updated list. Times are in nanoseconds (ns), microseconds (μs), or milliseconds (ms). Reminder: 1 ms = 1,000 μs = 1,000,000 ns.

Let me show you the same data visually, on a logarithmic scale so the orders of magnitude are obvious:

flowchart TD
    L1["L1 cache<br/>0.5 ns"]
    L3["L3 cache<br/>15 ns"]
    MEM["DDR5 memory<br/>80 ns"]
    SSD["NVMe random read<br/>15 μs"]
    DC["Datacenter RTT<br/>250 μs"]
    HDD["HDD random read<br/>8 ms"]
    XR["US ↔ Asia RTT<br/>150 ms"]

    L1 -->|"30x"| L3
    L3 -->|"5x"| MEM
    MEM -->|"200x"| SSD
    SSD -->|"15x"| DC
    DC -->|"30x"| HDD
    HDD -->|"20x"| XR

    style L1 fill:#34d399,stroke:#10b981,color:#000
    style L3 fill:#34d399,stroke:#10b981,color:#000
    style MEM fill:#34d399,stroke:#10b981,color:#000
    style SSD fill:#fbbf24,stroke:#f59e0b,color:#000
    style DC fill:#fbbf24,stroke:#f59e0b,color:#000
    style HDD fill:#fb7185,stroke:#f43f5e,color:#000
    style XR fill:#fb7185,stroke:#f43f5e,color:#000

A diagram showing the latency ladder from fastest to slowest. L1 CPU cache at half a nanosecond is the fastest thing software can do. L3 cache is 30 times slower at 15 nanoseconds. DDR5 main memory is 5 times slower again at 80 nanoseconds. NVMe random read jumps another 200 times slower to 15 microseconds. A datacenter round trip is 15 times slower than that at 250 microseconds. An old-school HDD random read is 30 times slower at 8 milliseconds. And a round trip across the globe  - US to Asia  - is another 20 times slower at 150 milliseconds. The total span from the fastest operation to the slowest is about 300 million times. That ratio is what makes design decisions consequential. If you accidentally do something across a region that you could have done in memory, you've slowed your code down by hundreds of millions of times.

The full span - from L1 cache at 0.5 ns to a US-to-Asia round trip at 150 ms - is 300 million times. Three hundred million. That's the range of "how slow can a single operation be" in modern computing. Knowing where things sit on this ladder is what lets you make good design choices.

The Human-Scale Analogy

Numbers like nanoseconds and microseconds are abstract. Let me make them concrete by scaling them up to human time. Imagine L1 cache reference takes 1 second (instead of 0.5 ns - we're scaling up by 2 billion).

At that scale:

• L1 cache (0.5 ns → 1 sec). Reaching into your pocket for your phone.
• Main memory (80 ns → 2.7 min). Walking to the kitchen to grab a snack.
• NVMe SSD random read (15 μs → 8 hours). One work day.
• Datacenter round trip (250 μs → 5.5 days). Almost a week.
• HDD random read (8 ms → ~6 months). A whole semester of college.
• US ↔ Asia round trip (150 ms → ~9.5 years). Childhood to graduation.

When you look at it this way, the costs become viscerally clear. If you're holding data in memory, it's like reaching into your pocket. If you're going across the world to fetch it, it's like waiting for a child to grow up. You really, really want data to be in your pocket whenever possible.

This is also why caching is such a powerful technique. Moving data from "across the world" (database in another region) to "in your pocket" (memory in your process) is a 100-million-x speedup. Few software optimizations come close.

What These Numbers Tell You

Now the practical part. How do you actually USE these numbers? Three main applications.

1. Designing for Latency

When designing a system, you can sum up the latencies along the request path and estimate the total response time. For example:

• Client → load balancer: ~1 ms (local network)
• Load balancer → app server: ~0.5 ms (datacenter)
• App server checks cache (Redis): ~0.5 ms
• App server queries database: ~5 ms
• Total: ~7 ms inside your infrastructure
• Plus network round trip from client to your datacenter: ~30–100 ms depending on geography

So a "fast" web request from a client takes ~50–100 ms end-to-end, mostly because of the user's network. The "everything inside your infrastructure" part is usually 5–20 ms.

When someone complains that an API is slow, you can check: is the slowness in your infrastructure (your servers, your database, your network) or in the user's network? The latency numbers tell you what's normal.

2. Choosing Between Storage Layers

The latency numbers tell you immediately which storage layer fits which use case:

• Need answer in < 1 μs? Must be in-process memory. No network call possible.
• Need answer in < 1 ms? Can hit a cache on the local network (Redis, Memcached). Database is too slow.
• Need answer in < 10 ms? Can hit a database. Cross-region call is too slow.
• Need answer in < 100 ms? Can do cross-region. But not many of them in series.
• Need answer in < 1 second? Can do quite a bit. But chains add up; budget carefully.

This gives you a sanity check on any design. "We're calling 5 services in series, each 30 ms - that's 150 ms total. Plus 50 ms for the user's network. We're at 200 ms response time." Now you know whether 200 ms is fast enough or whether you need parallelism / caching / a different design.

3. Estimating What Fits Where

A single Postgres machine can serve maybe ~10,000 simple queries per second from disk. (We'll get to throughput numbers in the next lesson.) But if those queries can be served from a cache instead - sub-millisecond - a single Redis instance can handle ~100,000+ queries per second. The latency difference is also a throughput difference.

Similarly: if you can keep your hot data in memory (using 100 GB of RAM, say), instead of going to disk, your serving capacity goes up by 100–1000x. This is why "fit it in RAM" is such a winning strategy for high-throughput systems. The latency numbers are why.

A Worked Example

Let me show you how to use the numbers in an actual design conversation.

Scenario: Someone proposes building a real-time leaderboard for a game. Every player completes a level, and the leaderboard updates. They want to read the leaderboard from anywhere in the world and have it appear in under 50 ms.

You ask: "Where will the leaderboard data live? And where will the read be served from?"

If the leaderboard data lives in a single primary database in one region (say US-East), then:

• A player in Asia reads the leaderboard
• Round trip to US-East: 150 ms
• Already over budget. Can't be done with this design.

To meet 50 ms, the data has to be served from somewhere geographically near every user. Options:

1. Replicate the data to every region (US-East, US-West, EU, Asia). Reads are local: ~5 ms. Writes are slower because they have to be propagated.
2. Cache the data at the edge (CDN with very short TTLs). Reads are very fast. Updates are slightly stale.
3. Use a globally-replicated database like Spanner or DynamoDB Global Tables. Reads from any region are fast; writes are slower.

Each option has trade-offs (we covered some in Module 2). But notice how the conversation became concrete the moment you used the latency numbers. Without them, you'd be debating in vibes; with them, you can see clearly that "50 ms from Asia" is impossible with a single US database, and the design has to change.

This is what these numbers are for. They take abstract performance discussions and turn them into concrete constraints.

Tips for Internalizing These

A few practical suggestions:

Don't try to memorize all 18 numbers at once. Instead, learn the orders of magnitude:

• Cache references: nanoseconds (single-digit to tens of ns)
• Memory: ~100 ns
• SSD random read: ~10 μs (NVMe) or ~100 μs (SATA)
• Datacenter network round trip: ~200–500 μs
• HDD random read: ~10 ms
• Cross-continent network round trip: ~50–150 ms

These six numbers are enough for 95% of practical estimation. The specific microsecond values you can look up; the relative scale you have to know.

Practice using them in conversation. Every time you read about a system, mentally check the latencies. "They're doing 10 database reads per request - that's ~50 ms. Plus the network - ~150 ms total. Seems consistent with their reported P95 of 200 ms."

Notice the 100x ratios. Each major step is roughly 100x: memory → local network → SSD → cross-region. Internalize that ratio. Then a question like "is doing this in-memory vs over the network going to matter?" answers itself: yes, by ~100x.

Don't memorize the cache hierarchy unless you do systems work. L1 vs L2 vs L3 mostly matters for low-level performance engineering. For HLD, just knowing "CPU caches are in nanoseconds, much faster than memory" is enough.

Six Takeaways from This Lesson

1. Latency numbers turn vibes into reasoning. They let you compare alternatives by performance, not by gut feel.

2. The order-of-magnitude ladder is what matters most. Cache → memory → SSD → local network → HDD → cross-region - each step is roughly 100x slower.

3. Memory is ~100 ns. Local network is ~250 μs. Cross-region is ~50–150 ms. Memorize these three; everything else is interpolation.

4. Speed of light limits cross-region latency. It's not getting faster. ~150 ms US to Asia is physics, not engineering.

5. The full span from L1 cache to cross-region is 300 million x. Where data lives on this ladder is one of the most consequential design choices you make.

6. Use the numbers in every design conversation. When someone proposes an architecture, run the latencies. The math often reveals issues that words alone hide.

Try It

Exercise: Warmup (10 minutes). Without looking back at the table: roughly how long does each of these take? (a) L1 cache reference, (b) main memory reference, (c) NVMe SSD random read, (d) datacenter round trip, (e) US-to-Asia round trip. Check your answers; note which you got wrong.

Exercise: Core (20 minutes). For an app you've worked on, estimate the latency contribution of each step in a typical request: network from user to server, request parsing, database query, response serialization, network back. Where is the time mostly spent?

Exercise: Stretch (30 minutes). Read the latest Google Cloud or AWS networking documentation. Find their published cross-region latency numbers (e.g., "us-east-1 to eu-west-1"). Compare to the numbers in this lesson. They should be in the right ballpark.

What's Next

Lesson 3.2 covers the other half: throughput numbers. Latency tells you how fast one operation goes; throughput tells you how many operations per second a system can sustain. Together, they're the two axes of capacity planning.

Back to System Design Crash Course