AlgoInsight

Free lesson · System Design Crash Course

Durability: Write Durability, Replication, and What "Lost" Means

Easy12 min readsystem-designhlddistributed-systems

Lesson 2.5 - Durability: Write Durability, Replication, and What "Lost" Means

The "saved" lesson.

Quick thought experiment. You're typing a document. You hit Ctrl-S. The status bar says "Saved." You smile and close the laptop. Later, the laptop battery dies before you got to plug it in. You boot it back up. Is your document still there?

Maybe. Probably. But not certainly. There's a non-zero chance that when the OS said "Saved," the data was actually still sitting in a write buffer in memory or on the disk controller, not yet committed to physical storage. If the power cut at exactly the wrong moment, you might lose work even though you saw the "Saved" message.

This sounds like a paranoid concern at the laptop level, and for most personal computing, it is. But scale this concern up - to a database storing financial transactions, to a cloud storage system holding millions of customers' photos, to a payment processor handling money - and the question of "when is data actually saved?" becomes one of the most important things to understand.

Durability is the dimension that captures this concern. It's about whether data, once written, survives the various ways it could be lost: power failures, disk failures, machine failures, datacenter failures. This lesson is about understanding the layers of durability, what "11 nines" means, and how systems achieve different durability guarantees.

What Durability Actually Means

Here's the working definition: durability is the probability that data, once accepted as "written," will still be retrievable later, no matter what fails.

Like availability, durability is usually expressed as a probability or as "nines." But the calculation is different. Availability is about being able to access the system right now. Durability is about not losing the data, period - even over a long horizon.

A system can have high availability but low durability (always responding fast, but loses data sometimes). A system can have low availability but high durability (frequently down, but never loses data). And of course, you want both, which is usually the engineering goal.

The standard claim from Amazon S3: 99.999999999% durability. That's 11 nines. The promise: if you store 10 million objects in S3, you'd expect to lose ONE object every 10,000 years.

Eleven nines is a wild number. Let me explain how anyone gets to that level, and why it's actually achievable.

The Journey of a Write

To understand durability, you need to understand what actually happens when you write data. Let me trace it through layers.

Suppose your application does this:

db.execute("INSERT INTO orders VALUES (...)")

What happens between that line of code and "the data is durable"?

flowchart TD
    A["1. Application calls db.execute()"]
    B["2. Database server receives over network"]
    C["3. Database writes to in-memory log buffer"]
    D["4. Database writes to write-ahead log (WAL)<br/>on local disk"]
    E["5. WAL is flushed to disk (fsync)"]
    F["6. Replication: WAL streamed to N replicas"]
    G["7. Replicas acknowledge"]
    H["8. Replica writes to their local WAL"]
    I["9. Backup system snapshots WAL"]
    J["10. Snapshots replicated cross-region"]

    A --> B --> C --> D --> E --> F --> G --> H --> I --> J

    style C fill:#fb7185,stroke:#f43f5e,color:#000
    style D fill:#fbbf24,stroke:#f59e0b,color:#000
    style E fill:#84cc16,stroke:#65a30d,color:#000
    style F fill:#34d399,stroke:#10b981,color:#000
    style J fill:#a78bfa,stroke:#8b5cf6,color:#000

A diagram tracing what happens when an application performs a database write. First, the application calls execute. Second, the database server receives the request over the network. Third, the database writes the change into an in-memory log buffer. Fourth, it writes that change to a write-ahead log on local disk. Fifth, the WAL is flushed to disk using fsync, ensuring it physically reaches durable storage. Sixth, the change is replicated to other database servers. Seventh, replicas acknowledge. Eighth, replicas also write to their own WAL. Ninth, a backup system takes snapshots of the WAL. Tenth, snapshots are replicated across regions. At each step, the data becomes more durable. A database can return "success" to the application at any of these steps  - and the choice has huge implications. Return at step 3 and you have very low durability. Return at step 5 and the data survives a server crash but not a disk loss. Return at step 7 and the data survives some replica failures. Return at step 10 and the data survives even a region-wide disaster.

The key insight: databases choose where to return "success" based on configurable durability settings. Faster acknowledgments mean less durability. Slower acknowledgments mean more.

Where Different Systems Acknowledge

Let me give concrete examples of how real systems handle this:

Naive in-memory database (Redis without persistence). Acknowledges at step 3 - data in memory. If the server crashes, data is gone. Used when speed matters more than durability (caches, session storage). Note: Redis can be configured for persistence, which changes this.

Postgres with default settings. Acknowledges at step 5 - data written to WAL on disk via fsync. Survives database server crash. But doesn't replicate synchronously by default - if the entire server's disk fails, data could be lost.

Postgres with synchronous replication enabled. Acknowledges at step 7 - data confirmed on at least one replica. Survives single-server disk failure.

Spanner / CockroachDB. Acknowledges only after consensus across multiple replicas. Strong durability but higher latency for writes.

S3. Acknowledges only after the object is durably stored across multiple availability zones. Plus erasure coding within zones. Plus periodic verification. The 11-nines durability comes from all of this layered.

Kafka with acks=all setting. Producer waits for confirmation from all in-sync replicas before considering the write complete. Higher durability than acks=1 (just the leader).

These choices are knobs. Most production systems let you tune them. The trade-off: faster writes = lower durability; slower writes = higher durability.

The fsync Story

Step 5 in our diagram - "WAL flushed to disk (fsync)" - deserves its own discussion because it's where a lot of durability bugs hide.

Modern operating systems do not write to disk immediately. When your code calls write(), the OS typically just puts the data in a kernel buffer and returns success. The actual disk write happens lazily, when the OS decides - could be milliseconds later, could be seconds later if the disk is busy.

For most applications, that's fine. Faster. The cost is that if the machine crashes between when you wrote and when the OS actually flushed to disk, your "successful" write is lost.

fsync() is the system call that forces the OS to actually write to disk before returning. Databases call it after every transaction (or every group of transactions in a batched mode) to ensure data is really on disk before they tell the application "success."

The catch: fsync is slow. On a spinning disk, it might take 10-20ms. On an SSD, sub-millisecond but still real time. On NVMe, faster but still measurable. This is why high-throughput databases batch transactions - group many writes into one fsync to amortize the cost.

It gets worse. fsync isn't even guaranteed to actually flush to disk in all cases. Some disk controllers have their own write caches. Some cheap SSDs lie about fsync (returning success before actually committing). Some virtualization layers buffer writes. The full story of "is this data actually on disk?" requires understanding the entire stack from application down to physical media.

Postgres has detailed documentation on this - the Reliability and the Write-Ahead Log page is required reading if you're ever responsible for a database. The summary: don't trust that fsync did what you think it did until you've verified the entire stack.

This is also why power-loss-protection (PLP) is a feature on enterprise SSDs but not consumer ones. PLP SSDs have onboard capacitors that flush in-flight writes during power loss. Consumer SSDs don't, which is why they shouldn't be used for databases.

Replication: The Multiplier

A single disk is not durable enough for serious workloads. Disks fail. Drives wear out. At Google's scale, drives fail constantly - multiple per day across their fleet.

The solution is replication: store multiple copies of data on different disks, on different machines, ideally in different physical locations.

The simplest form: write each piece of data to N machines simultaneously. If any single machine fails, the others still have the data. This is called "N-way replication."

Common replication factors:

Why is N=3 so common? Probability math.

Suppose one disk has a 1% chance of failure in a year. With three independent disks:

$P(\text{all 3 fail in the same year}) = 0.01^3 = 0.000001 = \text{1 in a million}$

That's already six nines of durability per year. And in practice the math is more complex because disks aren't exactly independent (correlated failures from common power, common hardware, etc.) and recovery happens fast (a failed disk gets replaced; a new replica gets built from the surviving copies).

flowchart LR
    Write["Write request"]

    subgraph Replicas["3-way replication"]
        R1["Replica 1<br/>(Server A)<br/>has data"]
        R2["Replica 2<br/>(Server B)<br/>has data"]
        R3["Replica 3<br/>(Server C)<br/>has data"]
    end

    Write --> R1
    Write --> R2
    Write --> R3

    R1 -.->|"if A fails, data on B and C"| R2
    R2 -.->|"if B fails, data on A and C"| R3

    style R1 fill:#34d399,stroke:#10b981,color:#000
    style R2 fill:#34d399,stroke:#10b981,color:#000
    style R3 fill:#34d399,stroke:#10b981,color:#000

A diagram showing 3-way replication. A write request is sent to three different replicas  - one on Server A, one on Server B, one on Server C. All three store the same data. If Server A fails, the data is still safe on B and C. If two servers fail simultaneously  - extremely rare  - only then is the data unavailable until recovery. This is the standard approach for production databases and is why three-way replication is the default in systems like Cassandra, MongoDB, and many others. The math: with independent failures, three replicas give roughly six nines of durability against single-disk failures per year.

For higher durability, systems go beyond simple N-way replication. They use erasure coding - a more efficient form of redundancy where data is split into chunks, additional parity chunks are computed, and the data can be reconstructed even if some chunks are missing. Erasure coding gives the same durability as N-way replication at a fraction of the storage cost - at the price of more compute (the parity calculation).

S3 uses erasure coding. So does HDFS (Hadoop's storage). So do most planet-scale storage systems. We cover erasure coding more in Module 17.

The Eleven Nines of S3

Let me decode the famous "99.999999999% durability" number from Amazon S3.

That's 11 nines of durability. Translated: if you store 10 million objects, you'd expect to lose ONE object every 10,000 years.

Or equivalently: 0.000000001% chance of losing any object in any given year.

How does S3 actually achieve this? The exact internal details are proprietary, but published information indicates:

The 11-nines number assumes all these protections are in place AND that you're using S3 Standard (cross-AZ). For S3 single-AZ (S3 One Zone-IA), the durability is "only" 99.999999999% per AZ - but if the AZ has a disaster, you lose data. So they don't sell that as 11 nines globally.

This is one of the most impressive engineering achievements in modern infrastructure. Brandon has direct experience with this since S3 is his domain at Amazon, and the engineering effort behind those eleven nines is enormous and ongoing.

The honest caveats:

Durability vs Availability (Important Distinction)

Worth emphasizing: durability and availability are NOT the same thing.

Durability: "Once you said it was saved, will it still be there if I ask in 5 years?"

Availability: "Can I get to it right now?"

A system can be high in one and low in the other:

These are different design concerns. The patterns for high durability (replication, persistence, erasure coding) are different from the patterns for high availability (redundancy, failover, graceful degradation). You can - and should - design for both, but as separate dimensions.

Durability Failure Modes (Real Examples)

Some real-world examples of how data gets lost, and what they teach:

GitLab, 2017. An engineer accidentally ran a destructive command on the production database. Backups failed silently (no one noticed they'd been failing for months). Recovery took 18 hours and lost about 6 hours of customer data. Lesson: backups must be verified regularly, not just configured.

Cloudflare, 2017. Memory corruption bug leaked customer data into other customers' responses. Not data loss, but a durability-adjacent integrity failure. Lesson: integrity checks throughout the stack matter.

Various crypto exchanges. Multiple have lost user funds due to inadequate durability (single-server wallets, no proper backups). Lesson: durability is a function of process, not just technology.

MyBB forums, 2020s. Many small forums have lost years of data when their host failed and they hadn't been making backups. Lesson: durability requires deliberate, redundant systems. "We have backups" should mean "we have tested backups in multiple locations."

The patterns:

  1. Backups that aren't verified are not backups.
  2. Single-location data is not durable, period.
  3. Replication isn't enough if replicas can be corrupted by the same bug.
  4. Operational mistakes are a major source of data loss, often more than hardware failures.

For any system holding important data, durability requires deliberate engineering: replication across machines, replication across regions, regular backup tests, immutable backup retention, monitoring of backup health, and operational procedures designed to prevent accidental destruction.

Practical Durability Levels

For different kinds of data, different durability levels are appropriate:

Critical financial data, user accounts, audit logs:

General business data, user content:

Caches, transient state:

Logs, metrics, analytics data:

The decision depends on the cost of losing the data vs the cost of higher durability. Don't pay for 11-nines durability on data you could regenerate from logs in a few hours. Don't skimp on durability for data that, if lost, would shut down the business.

Six Takeaways from This Lesson

  1. Durability is about whether data survives over time, in the face of failures. Different from availability, which is about right-now access.

  2. A write goes through many layers - memory, disk, fsync, replication, backup. Systems choose where to acknowledge "saved," and that choice determines durability.

  3. fsync is the line between "in OS buffer" and "actually on disk." Many durability bugs hide in misuse of fsync or in storage hardware that lies about it.

  4. N-way replication is the standard durability technique. N=3 is the common production choice, giving roughly six nines durability against single-disk failures.

  5. S3's 11 nines durability comes from multiple AZs, erasure coding within AZs, continuous integrity checking, and self-healing. It's a statistical claim, not a per-object guarantee.

  6. Durability requires deliberate engineering, not just technology choices. Backups must be verified. Multiple locations are required. Operational mistakes cause most data loss, not hardware failures.

Try It

Exercise: Warmup (10 minutes). For a system you maintain (or know well), trace one write request through the layers. Where does it acknowledge? What guarantees do you actually have?

Exercise: Core (25 minutes). Read Amazon S3's durability documentation and compare the durability commitments across storage classes (S3 Standard, S3 Intelligent-Tiering, S3 Glacier, S3 One Zone-IA). Note the difference between durability and availability.

Exercise: Stretch (45 minutes). Read the GitLab 2017 postmortem in full. Identify each failure that contributed to data loss. What durability practices would have prevented the worst impact?

What's Next

Lesson 2.6 introduces two famous frameworks for thinking about distributed system trade-offs: CAP and PACELC. You've probably heard of CAP. The version most people learned is slightly wrong. We'll fix that, then introduce PACELC, which is more useful for actual design decisions.

Back to System Design Crash Course