Free lesson · System Design Crash Course

The Full Hierarchy: Registers to Cold Storage

Medium12 min readsystem-designhlddistributed-systems

Lesson 7.1 - The Full Hierarchy: Registers to Cold Storage

The first lesson of Module 7.

Let me start with a question that sounds trivial. When your code says x = data[i], where does that value come from?

The naive answer: "from memory." But that hides everything interesting. The value might be in a CPU register (accessed in less than 1 nanosecond). It might be in L1 cache (~1 ns). It might be in L2 (~3 ns) or L3 (~10 ns). It might be in main memory (~80 ns). If your process has been swapped, it might be on disk (~100 microseconds for SSD, ~10 milliseconds for HDD). If your data is in a database on another machine, it could be milliseconds across the network. If it's in S3, tens of milliseconds. If it's archived to Glacier Deep Archive, it might take hours to retrieve.

All of these are "where the data is." The full spread is roughly a trillion times between the fastest (registers) and the slowest (deep cold storage). That's the storage hierarchy. This lesson walks down the entire ladder, level by level, so you have a clear mental model of where your data could live and what each level costs.

This isn't academic. The choice of where to store data is one of the most consequential architectural decisions you make. Knowing the hierarchy lets you make this choice deliberately, not by accident.

The Hierarchy In One Picture

Here's the full ladder, top to bottom:

flowchart TD
    Reg["CPU Registers<br/>~0.3 ns<br/>~1 KB capacity"]
    L1["L1 Cache<br/>~1 ns<br/>~64 KB per core"]
    L2["L2 Cache<br/>~3 ns<br/>~1 MB per core"]
    L3["L3 Cache<br/>~10-15 ns<br/>~32 MB shared"]
    RAM["Main Memory (DRAM)<br/>~80 ns<br/>1 GB to ~4 TB per server"]
    NVMe["NVMe SSD<br/>~10-100 μs<br/>~100 GB to 100 TB"]
    SATA["SATA SSD<br/>~100 μs<br/>~100 GB to 100 TB"]
    HDD["HDD (spinning disk)<br/>~5-15 ms<br/>~1-20 TB per drive"]
    Local["Local Network Storage<br/>~250 μs - 1 ms<br/>~TB to PB"]
    Object["Object Storage (S3)<br/>~10-100 ms<br/>Effectively unlimited"]
    Glacier["Cold Archive (Glacier Deep)<br/>Hours to retrieve<br/>Effectively unlimited"]

    Reg --> L1
    L1 --> L2
    L2 --> L3
    L3 --> RAM
    RAM --> NVMe
    NVMe --> SATA
    SATA --> HDD
    HDD --> Local
    Local --> Object
    Object --> Glacier

    style Reg fill:#34d399,stroke:#10b981,color:#000
    style L1 fill:#34d399,stroke:#10b981,color:#000
    style L2 fill:#34d399,stroke:#10b981,color:#000
    style L3 fill:#34d399,stroke:#10b981,color:#000
    style RAM fill:#34d399,stroke:#10b981,color:#000
    style NVMe fill:#fbbf24,stroke:#f59e0b,color:#000
    style SATA fill:#fbbf24,stroke:#f59e0b,color:#000
    style HDD fill:#fbbf24,stroke:#f59e0b,color:#000
    style Local fill:#fb7185,stroke:#f43f5e,color:#000
    style Object fill:#fb7185,stroke:#f43f5e,color:#000
    style Glacier fill:#a78bfa,stroke:#8b5cf6,color:#000

A diagram showing the full storage hierarchy from top to bottom. At the very top, CPU registers  - accessed in about a third of a nanosecond, but only about a kilobyte total per core. Just below, L1 cache at one nanosecond and 64 kilobytes per core. L2 cache at three nanoseconds and about a megabyte per core. L3 cache at 10-15 nanoseconds and about 32 megabytes shared across cores. These four are all silicon  - physically inside the CPU package. Then main memory, DRAM, at about 80 nanoseconds and gigabytes to terabytes. NVMe SSD at 10 to 100 microseconds and hundreds of gigabytes to tens of terabytes. SATA SSD slightly slower at 100 microseconds. HDD or spinning disk at 5 to 15 milliseconds  - a million times slower than registers. Network storage at hundreds of microseconds to milliseconds. Object storage like S3 at 10 to 100 milliseconds. And at the bottom, cold archive like Glacier Deep Archive which takes hours to retrieve. The full span is a factor of about a trillion in latency. Where your data sits on this ladder is one of the most consequential architectural decisions you make.

This is the ladder. Memorize at least the orders of magnitude. The specific numbers matter less than knowing the shape of the hierarchy.

The Three Properties That Vary

As you move down the hierarchy, three things change in lockstep:

1. Latency increases. Each level is 10-1000x slower than the one above. Total span from top to bottom is roughly 10^12 (a trillion times).

2. Capacity increases. Registers hold bytes. L1 holds kilobytes. RAM holds gigabytes. SSDs hold terabytes. Object storage is effectively unlimited.

3. Cost per byte decreases. Registers are essentially free (built into the CPU). RAM is ~$2-4 per GB. SSD is ~$0.10 per GB. HDD is ~$0.02 per GB. Cold storage is ~$0.001 per GB. Roughly a 1000x range from RAM to cold.

These three properties are tightly coupled because of physics and economics. Fast storage (registers, cache, RAM) requires fancy silicon close to the CPU. Cheap storage (HDDs, tape) requires bulk materials and tolerance of mechanical or slow access.

flowchart LR
    subgraph Top["Top of Hierarchy"]
        TF["Fast<br/>~ns"]
        TS["Small<br/>~KB-MB"]
        TC["Expensive<br/>~$$/GB"]
    end

    subgraph Bottom["Bottom of Hierarchy"]
        BF["Slow<br/>~minutes"]
        BS["Huge<br/>~PB"]
        BC["Cheap<br/>~¢/GB"]
    end

    style Top fill:#34d399,stroke:#10b981,color:#000
    style Bottom fill:#a78bfa,stroke:#8b5cf6,color:#000

A diagram showing how three properties vary together across the storage hierarchy. At the top: fast latency measured in nanoseconds, small capacity in kilobytes to megabytes, expensive at dollars per gigabyte. At the bottom: slow latency potentially minutes, huge capacity in petabytes, cheap at cents per gigabyte. Latency, capacity, and cost are inversely correlated. Faster always means smaller and more expensive. Cheaper always means larger and slower. This isn't a coincidence of the market  - it's physics. To make storage faster you need it closer to the CPU and made of expensive materials. To make storage cheaper you accept slower access and use bulk materials. The economic and physical constraints align. This is why "just put everything in memory" doesn't work at scale  - you'd need petabytes of memory at thousands of dollars per gigabyte. Hierarchy is mandatory, not optional.

The hierarchy isn't an arbitrary design choice; it's the only way to handle data at scale economically. You can't put a petabyte in RAM. You can't put 16 GB on a tape cartridge that's worth keeping. Different volumes of data go in different places by necessity.

What Each Level Is

Let me give you a one-paragraph orientation to each level. We'll go into depth on each in subsequent lessons.

CPU Registers

The fastest storage. Built directly into the CPU. Used for the few values the CPU is actively operating on. A modern CPU has dozens of general-purpose registers per core; total capacity is measured in bytes (not even kilobytes). When the CPU does add a, b, both a and b are in registers. You don't manage registers directly in application code - the compiler decides which values go in registers. But the existence of registers is why some code patterns are faster than others.

CPU Caches (L1, L2, L3)

Built into the CPU package (the chip itself). L1 is closest to each core and smallest; L3 is shared across cores and largest. They cache recently-accessed memory so the CPU doesn't have to wait for slower DRAM. Modern CPUs have all three levels. Cache hits are dramatically faster than cache misses - sometimes by 100x. The caches are managed by hardware; software doesn't directly control them, but memory access patterns dramatically affect cache hit rates. We'll cover this in Lesson 7.2.

Main Memory (DRAM)

What you usually mean by "RAM." Volatile (loses data when power off), accessed in roughly 80 nanoseconds, sized in gigabytes to terabytes per server. Modern servers can have 1-4 TB of RAM; consumer machines have 16-128 GB. This is where most of your "data" lives during program execution - variables, allocated objects, loaded files, caches in software like Redis. Lesson 7.3 covers RAM in depth.

NVMe SSD

The current fastest persistent storage. Uses the NVMe protocol over PCIe lanes. Latency around 10-100 microseconds; throughput of multiple gigabytes per second. Capacity per drive from hundreds of GB to ~100 TB. This is what modern databases run on. Modern data center NVMe drives have remarkable performance - millions of IOPS per drive. Lesson 7.4 covers SSDs in depth.

SATA SSD

The previous-generation SSD interface. Uses the SATA protocol, which is the older drive interface. Slower than NVMe (~100 microseconds, ~500 MB/s sequential) but still much faster than HDD. Cheaper per GB than NVMe. Common in less performance-critical scenarios.

HDD (Spinning Disk)

Spinning magnetic platters with a mechanical read/write head. Latency 5-15 milliseconds for random reads (because the head has to physically move). Sequential throughput is decent (~200 MB/s) but random IOPS are abysmal (~100 per drive). Used for bulk storage where access patterns are sequential or where cost matters more than speed. Lesson 7.5 covers HDDs.

Network-Attached Storage

Storage that lives on a different machine, accessed over a network. Examples: AWS EBS (block storage), AWS EFS (NFS file system), NetApp filers in enterprise. Latency is dominated by the network - typically 250 microseconds to a few milliseconds. Throughput depends on the network link. Used when you want storage decoupled from the compute (so the compute can be replaced without data loss). Lesson 7.6.

Object Storage

Storage accessed via HTTP-style APIs rather than filesystem or block protocols. Examples: AWS S3, Google Cloud Storage, Azure Blob. Latency typically 10-100 milliseconds (HTTP request overhead). Effectively unlimited capacity. Designed for durability and scale - files (objects) are replicated automatically. Used for static content, backups, data lakes, large media files. The most-used storage at internet scale.

Cold Archive Storage

Object storage with retrieval latency in hours. Examples: AWS S3 Glacier Deep Archive, Azure Archive. Cost is roughly 100x cheaper than warm object storage but you wait for retrieval. Used for things you need to keep but rarely access - compliance archives, old backups, regulatory data.

This is the full ladder. In any system, your data lives at one or more of these levels.

Real Cost Comparison

Let me give you 2026 pricing (rough - varies by vendor and contract):

The dynamic range from RAM to deep archive is about 4000x in cost. For storage volumes that grow beyond gigabytes, this matters enormously. Storing 1 PB in RAM is essentially impossible economically. Storing 1 PB in cold archive is a few thousand dollars a month.

This is why tiering is universal in real systems. Hot data (frequently accessed) lives in fast tiers; cold data (rarely accessed) gets moved to cheaper tiers. The cost difference is too large to ignore.

Latency and Throughput at Each Level

For practical capacity planning, knowing latency isn't enough - you also need throughput. Here's an overview (from Module 3, Lessons 3.1 and 3.2):

Two things to notice:

1. Throughput drops more slowly than latency increases. From L1 to disk, latency increases by 10 million times. But throughput only drops by about 100 times. This is why sequential access from disk can be reasonable for large bulk operations, even though random access is terrible.

2. Modern NVMe is remarkable. A single drive can read at 14 GB/sec - that's almost as fast as DRAM bandwidth! For sequential workloads, NVMe approaches memory speeds. Random access is still slower, but the gap is much smaller than it used to be.

Why Locality Matters

A key principle that runs through the entire hierarchy: locality of reference.

• Temporal locality: if you accessed X recently, you'll probably access it again soon.
• Spatial locality: if you accessed X, you'll probably access something near X next.

This is why caching works. Hardware caches (CPU L1/L2/L3) bet on these patterns: load some memory, keep it close, the program will probably use it again. Software caches do the same. Operating systems do it for disk. Object stores do it via CDN edges.

If your code has good locality, every level of the hierarchy helps you. If your code has bad locality (random access patterns, jumping all over memory), every level hurts you. Cache misses, page faults, disk seeks all happen.

Designing for locality is one of the most underappreciated performance levers. We'll see specific patterns in Lesson 7.2.

A Mental Model: The Library

A useful analogy:

• Registers = the book you're currently reading
• L1 cache = the stack of books on your desk
• L2 cache = the books on the shelf next to your desk
• L3 cache = the books in your office bookshelf
• RAM = your personal home library
• SSD = the local library
• HDD = the regional library
• Network storage = the central library system
• Object storage = a remote archive you can request books from
• Cold archive = the offsite warehouse that takes a day to retrieve from

When you need a fact, you check the book in front of you first (fast). If it's not there, you check the desk (still fast). Then the shelf. Then your home. Eventually you might have to request from the warehouse, which takes a day.

Most facts you need recently are still close at hand. Rare facts require trips. Your job, as the designer of the system, is to keep frequently-used things close and tolerate the trips for rarely-used things.

How Real Systems Use the Hierarchy

Almost no real system uses just one level. Examples:

A typical web application:

• App code in RAM (loaded at startup)
• Active request data in CPU caches
• Session data in Redis (RAM on another server)
• Persistent data in Postgres (NVMe SSD)
• User-uploaded files in S3 (object storage)
• Cold backups in Glacier (cold archive)

Five different storage levels for one application, each fitting the data's access pattern.

A database engine internally:

• Index pages cached in RAM (in-memory caching of disk content)
• Recent transactions in WAL on NVMe (fast persistence)
• Older data on cheaper SSD or HDD
• Archived partitions on object storage

A database is itself a hierarchy-aware system.

A CDN:

• Hottest content in edge caches (RAM at the edge)
• Less-hot content in regional caches (SSD at regional points)
• Cold content fetched from origin (object storage)

The CDN is essentially a globally-distributed cache hierarchy.

These patterns are everywhere because the hierarchy is unavoidable. Working with it deliberately is how you build performant systems; ignoring it is how you build slow ones.

A Story From Practice

When I joined the S3 team, one of the first things I learned was how deeply S3 itself uses the hierarchy. S3 doesn't just dump data on disk - it's a massive multi-tier system.

When you upload an object, S3 stores it across many physical drives (HDDs, SSDs, depending on storage class) for durability. But the metadata about where everything is - bucket listings, ACLs, location maps - that lives in massive indexed key-value stores running on NVMe SSDs with aggressive RAM caching. When you query S3, the metadata path is in memory; the data path goes to disk.

When you read an object, S3 might:

1. Check RAM caches for the metadata
2. Hit the metadata service (SSD-backed)
3. Read the actual object data from disk (HDD for standard, NVMe for some tiers)
4. Stream it back over the network

Every layer of the hierarchy is in play. And the choice of which data lives where is a constant engineering optimization. Most-accessed metadata goes to memory; mid-frequency stuff goes to SSD; bulk archives go to HDD or even tape (for Glacier Deep Archive).

For application developers, the takeaway: even when you're using a "simple" service like S3, behind the scenes there's a sophisticated hierarchy doing the work. Understanding the hierarchy helps you reason about why S3 has the latency characteristics it has, why some patterns are cheap and others expensive, and where the trade-offs are.

This kind of understanding is what distinguishes "uses systems" from "designs systems."

Six Takeaways from This Lesson

1. The storage hierarchy spans 12 orders of magnitude in latency from CPU registers to cold archive. Knowing the ladder is essential.

2. Three properties move together: latency, capacity, cost. Faster always means smaller and more expensive.

3. The hierarchy exists for physics and economics reasons. It's not a design choice; it's mandatory at any meaningful scale.

4. Tiering is universal in real systems. Hot data in fast tiers; cold data in cheap tiers. Cost differences are too large to ignore.

5. Locality of reference makes the hierarchy work. Designing for good locality dramatically improves performance.

6. Most systems use multiple levels simultaneously. Understanding which level holds what is fundamental to reasoning about system performance.

Try It

Exercise: Warmup (10 minutes). For a typical web app you've worked on, identify where each kind of data lives. Application code? Session data? Database rows? Uploaded files? Backups? Mark them on the hierarchy.

Exercise: Core (20 minutes). Estimate the cost of storing 100 TB of data: (a) all in RAM, (b) all in NVMe SSD, (c) all in S3 Standard, (d) all in Glacier Deep Archive. The numbers reveal why tiering matters.

Exercise: Stretch (40 minutes). Read about how a database engine (Postgres, MySQL, or RocksDB) uses the storage hierarchy. Look at concepts like buffer cache, write-ahead log, page eviction. Note how the database is itself a hierarchy-aware system.

What's Next

Lesson 7.2 zooms in on the top of the hierarchy: CPU caches. Why memory access patterns matter so much for hot-path performance, and what application developers can do about it.

Back to System Design Crash Course