Free lesson · System Design Advanced

PACELC Theorem - The Refinement of CAP

Hard7 min readsystem-designadvanced

CAP theorem says: during a network partition, choose between consistency and availability. This is true, important, and almost completely useless for day-to-day system design decisions. Here's why: network partitions are rare events. A well-run AWS deployment might experience a meaningful partition once every few months. But the trade-off between latency and consistency happens on every single request, millions of times per day.

PACELC, proposed by Daniel Abadi in 2012, fills this gap. It says: during a Partition, choose between Availability and Consistency (that's the PAC - same as CAP). Else, when the system is running normally, choose between Latency and Consistency (that's the ELC). It's the "Else" clause that makes PACELC useful. It forces you to articulate what your system does in the common case - not just the failure case.

CAP's Blind Spot

graph TD
    subgraph "CAP Only Answers One Question"
        CAP_Q["During a network partition,<br/>what does the system do?<br/>──────────────────<br/>CP: reject requests to stay consistent<br/>  (DynamoDB strong reads, Spanner)<br/><br/>AP: serve requests with possibly<br/>  stale data to stay available<br/>  (DynamoDB eventual reads, Cassandra)<br/><br/>This matters maybe 0.01% of the time"]
        
        MISSING["The Question CAP Doesn't Answer<br/>──────────────────<br/>The other 99.99% of the time —<br/>when the network is FINE —<br/>what trade-offs does the system make?<br/><br/>Does it replicate synchronously<br/>(consistent but slow)?<br/>Or asynchronously<br/>(fast but potentially stale)?<br/><br/>CAP says NOTHING about this"]
    end

This diagram shows CAP's blind spot. CAP only answers what happens during a network partition - CP systems reject requests for consistency while AP systems serve possibly stale data for availability. But this matters maybe 0.01% of the time. The other 99.99% when the network is fine, CAP says nothing about the latency-versus-consistency trade-off that affects every single request.

PACELC: The Full Picture

graph TD
    subgraph "PACELC Decision Framework"
        START["Is there a network partition?"]
        
        START -->|"YES<br/>(rare event)"| PAC["PAC: Partition Case<br/>──────────────────<br/>Choose between:<br/><br/>A - Availability<br/>  Continue serving requests<br/>  Accept stale/inconsistent data<br/><br/>C - Consistency<br/>  Reject or block requests<br/>  Maintain correctness"]
        
        START -->|"NO<br/>(normal operation)"| ELC["ELC: Else Case<br/>──────────────────<br/>Choose between:<br/><br/>L - Latency<br/>  Replicate asynchronously<br/>  Respond before replication<br/>  completes (fast, might be stale)<br/><br/>C - Consistency<br/>  Replicate synchronously<br/>  Wait for replication before<br/>  responding (slow, always fresh)"]
    end

This diagram shows the PACELC framework. It starts by asking whether there's a network partition. If yes, choose between availability and consistency - this is the PAC case, same as CAP. If no, which is normal operation, choose between latency and consistency - this is the ELC case. Low latency means asynchronous replication where the system responds before replication completes. Consistency means synchronous replication where the system waits for replication before responding.

Real Systems Classified by PACELC

This is where PACELC becomes genuinely useful. Every database you'll encounter in a system design interview makes a specific PACELC choice, and understanding that choice tells you far more than knowing whether it's "CP" or "AP":

graph TD
    subgraph "PA/EL Systems (Availability + Low Latency)"
        PAEL["DynamoDB (eventual reads)<br/>Cassandra (default config)<br/>CouchDB<br/>──────────────────<br/>During partition: stay available<br/>During normal ops: low latency<br/>(async replication)<br/><br/>Philosophy: speed and availability<br/>always, consistency when possible<br/><br/>Trade-off: you might read<br/>stale data even when nothing<br/>is broken - by design"]
    end
    
    subgraph "PC/EC Systems (Consistency Always)"
        PCEC["Spanner<br/>CockroachDB<br/>FoundationDB<br/>──────────────────<br/>During partition: stay consistent<br/>During normal ops: stay consistent<br/>(sync replication, TrueTime/HLC)<br/><br/>Philosophy: correctness always,<br/>even at the cost of latency<br/><br/>Trade-off: every write waits<br/>for synchronous replication<br/>across zones/regions"]
    end
    
    subgraph "PA/EC Systems (Available During Partition, Consistent Normally)"
        PAEC["MongoDB (w:majority, readConcern:majority)<br/>PostgreSQL (sync replication)<br/>MySQL (semi-sync replication)<br/>──────────────────<br/>During partition: stay available<br/>(degrade to async, accept risk)<br/>During normal ops: stay consistent<br/>(synchronous replication)<br/><br/>Philosophy: be correct when things<br/>are working, degrade gracefully<br/>when they aren't<br/><br/>This is the most common<br/>production choice"]
    end
    
    subgraph "PC/EL Systems (Consistent During Partition, Fast Normally)"
        PCEL["Rare in practice<br/>──────────────────<br/>During partition: reject requests<br/>During normal ops: low latency<br/>(async replication)<br/><br/>Unusual combination: willing to<br/>go unavailable during partition<br/>but doesn't bother with sync<br/>replication normally<br/><br/>Hard to justify in practice"]
    end

This diagram classifies real databases by their PACELC choices. PA/EL systems like DynamoDB with eventual reads and Cassandra prioritize availability during partitions and low latency during normal operations - you might read stale data by design. PC/EC systems like Spanner and CockroachDB prioritize consistency always, both during partitions and normal operations, at the cost of latency. PA/EC systems like MongoDB with majority concerns and PostgreSQL with sync replication are the most common production choice - consistent during normal operation but degrading to availability during partitions. PC/EL systems are rare because the combination of rejecting requests during partitions while using async replication normally is hard to justify.

Interview: When an interviewer asks "is this system CP or AP?" the strong answer goes beyond CAP. Say: "CAP only describes partition behaviour. Under normal operation, this system is EC - it uses synchronous replication, so every read sees the latest write, but at the cost of higher write latency. The full PACELC classification is PA/EC: during a partition it degrades to availability, but in steady state it prioritizes consistency." This demonstrates you understand that the common-case trade-off matters more than the failure-case trade-off.

The Latency-Consistency Spectrum

The ELC part of PACELC creates a continuous spectrum, not a binary choice:

Important: The most common mistake in system design interviews is treating consistency as binary - "consistent" or "eventually consistent." In reality, it's a spectrum with at least five meaningful positions. Each position has a different latency cost and a different failure mode. The engineer who says "I'd use semi-synchronous replication to get majority-acknowledged durability with ~3ms of overhead, which is acceptable for our 50ms latency budget" is demonstrating a fundamentally different level of understanding than the one who says "I'd make it consistent."

The Production War Story

At Amazon, I worked on a service that replicated order data across three availability zones. The original design used fully synchronous replication - every write waited for all three AZs to confirm before acknowledging. This was a PC/EC design: consistent during partitions (reject writes if a zone is unreachable) and consistent during normal operation (wait for all replicas).

The consistency guarantee was perfect. The latency was not. Median write latency was 12ms, which was fine. But during peak traffic, cross-AZ latency would spike to 40-80ms due to network congestion. P99 write latency hit 120ms. The order submission flow had a 200ms latency budget for the entire pipeline, and the database was consuming 60% of it just for replication.

We redesigned as PA/EC: semi-synchronous replication that waited for two of three AZs (majority). During normal operation, this was still consistent - a majority acknowledgment means the write survives any single-AZ failure. During a partition (one AZ isolated), the system continued serving with two AZs, degrading to eventual consistency for the third.

The result: median write latency dropped from 12ms to 7ms (only waiting for the fastest of two replicas, not all three). P99 dropped from 120ms to 25ms (no longer bottlenecked on the slowest AZ). The consistency guarantee was weaker on paper (majority vs all) but in practice, we had the same durability - a single AZ failure was survivable with both designs. We just stopped paying latency tax for the two-AZ-simultaneous-failure scenario that had never happened in the service's three-year history.

That's the PACELC insight in a single story: understanding what you're actually trading - not "consistency vs availability" in the abstract, but "3ms of P50 latency and 95ms of P99 latency in exchange for surviving a failure mode that has literally never occurred."

Six Takeaways

1. CAP only describes partition behaviour; PACELC adds the common case. Since partitions are rare, the ELC trade-off (latency vs consistency during normal operation) is the decision you make millions of times per day.

2. PA/EC is the most common production choice. Systems like MongoDB with majority concerns, PostgreSQL with synchronous replication, and MySQL with semi-sync prioritize consistency during normal operation but degrade to availability during partitions.

3. The latency-consistency trade-off is a spectrum, not a binary. From fire-and-forget (0ms, no guarantee) through semi-sync (~3ms, majority-committed) to cross-region synchronous (~100ms, globally consistent) - each position has a cost and a failure mode.

4. PC/EC systems exist but are expensive. Spanner and CockroachDB provide consistency always, but you pay in latency on every write. This is justified for financial systems, not for most CRUD applications.

5. Synchronous replication to ALL replicas is almost never worth it. Majority-based replication (2 of 3) gives you the same practical durability at significantly lower latency by not waiting for the slowest replica.

6. Name the PACELC classification in interviews. "This system is PA/EC - available during partitions, consistent during normal operation using semi-synchronous replication" is a sentence that demonstrates real distributed systems understanding.

Exercises

Exercise: Starter: Classify each of these real systems by their PACELC behaviour: (a) Amazon DynamoDB with eventual consistency reads. (b) Amazon DynamoDB with strongly consistent reads. (c) PostgreSQL with synchronous_commit = on and one synchronous replica. (d) Redis with async replication. (e) Google Spanner. For each, explain what happens during normal operation AND during a network partition.

Exercise: Intermediate: You're designing a global shopping cart service deployed across US-East, US-West, and EU-West. Write latency budget: 100ms. Read latency budget: 50ms. (a) A PC/EC design uses synchronous replication across all three regions. Estimate the write latency (hint: US-East to EU-West is ~80ms round-trip). Does this fit the budget? (b) A PA/EC design uses synchronous replication within-region and async cross-region. Estimate write and read latency. What happens when a user adds an item in US-East and reads their cart from EU-West 200ms later? (c) A PA/EL design uses async replication everywhere. What are the latency characteristics? What failure modes does the user experience? (d) Recommend the right PACELC classification for this workload and defend your choice.

Exercise: Advanced: Your company operates a financial ledger service that must never lose a committed transaction and must serve globally with sub-100ms reads. (a) Argue for PC/EC (Spanner-like). What's the write latency cost? Is it acceptable for financial transactions? (b) Argue for PA/EC with cross-region async and within-region sync. What failure scenarios can cause data loss? How would you mitigate each? (c) Design a hybrid: PA/EC for the primary write region with async replication to read-only replicas in other regions. How does this affect the PACELC classification? Is it PA/EC or something that doesn't fit neatly into the framework? (d) Critique PACELC itself: what trade-offs does the framework fail to capture? (Hint: think about read vs write paths, per-operation tuning, and multi-tenancy.)

Back to System Design Advanced