Free lesson · System Design Fundamentals

Reliability

Easy8 min readsystem-designhlddistributed-systems

Lesson 2.3 - Reliability

The third lesson of Module 2.

Availability and reliability sound interchangeable. They're not, and the difference has caused more production incidents than most engineers realize. Availability answers: "is the system responding?" Reliability answers: "is it responding correctly, consistently, and predictably over time?" A system can be 100% available - responding to every single request - and simultaneously unreliable if 3% of those responses contain wrong data, corrupt a write, or silently drop a message. The server is up. The answers are wrong. That's available but unreliable, and it's more dangerous than being simply down - because nobody notices until the damage is done.

The Difference, Concretely

Consider five scenarios for a payment processing system:

The second and fourth rows are the truly dangerous cases. The system looks healthy on dashboards - uptime is 100%, error rate is 0%. But the data is wrong. These failures are silent - they don't trigger availability alerts because the server is responding. They surface weeks later during financial reconciliation, customer complaints, or audit - by which point thousands of incorrect transactions may have occurred.

This is why reliability is a separate concept from availability. Monitoring that only tracks "is the server responding?" misses the most damaging failure mode: "the server is responding incorrectly."

The Metrics of Reliability

Two metrics quantify how systems fail and recover:

MTBF - Mean Time Between Failures. The average time the system operates correctly before the next failure. A system with an MTBF of 720 hours (30 days) experiences a failure roughly once per month. Higher MTBF means failures are rarer. You improve MTBF through better testing, more robust code, higher-quality hardware, simpler architectures (fewer moving parts = fewer things that can break), and chaos engineering (finding weaknesses before they cause real failures).

MTTR - Mean Time To Recovery. The average time from when a failure is detected to when the system is fully operational again. A system with an MTTR of 15 minutes recovers quickly from each failure. Lower MTTR means faster recovery. You improve MTTR through automated detection (monitoring, alerting - M43), automated remediation (auto-restart, auto-failover), runbooks (documented steps for common failures), and practice (game days, incident drills).

flowchart LR
    N1["✅ Normal\noperation"]
    F["❌ Failure\noccurs"]
    D["🔍 Detection\n(monitoring alerts)"]
    R["🔧 Recovery\n(fix, failover, restart)"]
    N2["✅ Normal\noperation"]
    F2["❌ Next\nfailure"]
    
    N1 -->|"MTBF\n(time between failures)"| F
    F -->|"detection\ntime"| D
    D -->|"repair\ntime"| R
    R --> N2
    N2 -->|"MTBF"| F2
    
    F -.->|"MTTR\n(detection + repair)"| R

    style F fill:#fb7185,stroke:#f43f5e,color:#000
    style D fill:#fbbf24,stroke:#f59e0b,color:#000
    style R fill:#34d399,stroke:#10b981,color:#000

A timeline diagram showing the reliability lifecycle. The system operates normally for a period - that's the MTBF, mean time between failures. Then a failure occurs. The failure must first be detected - via monitoring alerts, health checks, or user reports. Detection takes time, especially for silent failures. Then the system must be repaired - through automated failover, manual restart, code fix, or data correction. The time from failure occurrence to full recovery is the MTTR, mean time to recovery. MTTR includes both detection time and repair time. After recovery, normal operation resumes until the next failure. Availability can be approximated as MTBF divided by MTBF plus MTTR. A system that fails every 30 days (MTBF = 720 hours) and recovers in 15 minutes (MTTR = 0.25 hours) has availability of 720 divided by 720.25, which equals 99.97 percent. The critical insight: there are two strategies for improving availability. Strategy one: increase MTBF - fail less often. Better testing, better hardware, simpler architecture. Strategy two: decrease MTTR - recover faster. Automated detection, automated failover, runbooks, practice drills. For most teams, reducing MTTR is higher leverage than increasing MTBF. You cannot prevent all failures - hardware dies, networks partition, humans make mistakes. But you CAN make recovery fast and automated. Investing in fast recovery (monitoring, auto-failover, runbooks) has a higher return than investing in preventing all failures (which is impossible).

The relationship between MTBF, MTTR, and availability:

Availability ≈ MTBF / (MTBF + MTTR)

Notice: reducing MTTR from 4 hours to 15 minutes (a 16x improvement in recovery speed) improves availability more than increasing MTBF from 30 days to 365 days (a 12x improvement in failure frequency). Fast recovery beats rare failures. This is the core operational insight of reliability engineering.

Fault Tolerance vs Fault Avoidance

Two complementary strategies:

Fault avoidance tries to prevent failures from happening. Code reviews catch bugs. Automated testing catches regressions. Capacity planning prevents overload. Hardware redundancy prevents single-machine failures. Fault avoidance is necessary - a system with no testing, no reviews, and no redundancy will fail constantly. But it's insufficient - you cannot prevent every failure. Hardware dies. Networks partition. Humans make mistakes. Dependencies fail.

Fault tolerance assumes failures WILL happen and designs the system to continue working correctly despite them. A replicated database (M28): one replica dies, the others serve reads and a new replica is promoted. Retry with idempotency (M42): a request fails, the client retries safely. Circuit breakers (M42): a dependency dies, the system degrades gracefully instead of cascading. A message queue (M9): the consumer crashes, messages are retained and processed when it recovers.

Reliable systems are fault-tolerant, not fault-free. They fail - but they fail gracefully, they recover automatically, they preserve data integrity, and they communicate their state clearly. Module 42 (Reliability Patterns) is entirely about building fault tolerance into every layer.

Correctness: The Hardest Reliability Property

Availability failures are loud - the server is down, alerts fire, users complain. Correctness failures are silent - the server is up, responses look normal, but the data is subtly wrong.

Categories of correctness failures:

Data corruption. A race condition in concurrent writes produces garbled data. A rounding error in financial calculations produces slightly wrong amounts. A character encoding mismatch produces corrupted text. These are the failures that checksums (M3.6), ACID transactions (M14), and formal verification catch - when they're implemented.

Data loss. A write is acknowledged to the client but never reaches durable storage (the server crashes between the acknowledgment and the disk flush). A replication lag (M28) means a failover loses the last 5 seconds of writes. These are the failures that write-ahead logs (M14), synchronous replication (M28), and exactly-once semantics (M34) prevent.

Stale data. A read returns an old version of the data because it hit a replica that hasn't received the latest write. The data isn't wrong - it's outdated. This is the consistency model spectrum (2.8) - strong consistency prevents staleness but costs latency; eventual consistency allows staleness but is faster.

Ordering violations. Events are processed in the wrong order - a refund is processed before the charge it refunds, a message reply appears before the message it replies to. Stream processing (M34) and event ordering (M27 - Time and Ordering) address these.

A Brandon Story: Available But Wrong

A team I'd advised ran a financial reporting system for an investment platform. Availability was outstanding - 99.98% uptime over 18 months, no outages longer than 5 minutes. The monitoring dashboards were green. The on-call team was bored.

During a database failover event (planned maintenance, primary-to-replica promotion), a race condition in the replication caused 0.2% of write transactions to be recorded with slightly incorrect amounts. The mechanism: during the 3-second failover window, some writes landed on the old primary (which was being demoted) and the new primary (which was being promoted), with slightly different timestamps producing different rounding behavior in currency conversion calculations. Each affected transaction was off by fractions of a cent - individually invisible, collectively adding up to

The system was available the entire time. No alerts fired. The monitoring tracked response times and error rates - both normal. Nobody noticed for three weeks, until the monthly financial reconciliation flagged the discrepancies. By that point, 400 transactions had incorrect amounts in the system of record. Each required manual correction, regulatory documentation, and customer notification. The cleanup took 6 weeks and involved the legal team.

The root cause wasn't the failover - it was the testing. The failover was tested for availability: "does the system stay up during failover?" Yes. It was NOT tested for reliability: "do all writes produce correct results during failover?" No. The write-path correctness during the 3-second transition window had never been verified.

The fix:

1. Write verification - every write is read back within 5 seconds and compared. Mismatches trigger immediate alerts. MTTR for correctness failures dropped from "3 weeks" (discovered during reconciliation) to "5 seconds" (discovered by automated verification).
2. Failover correctness testing - quarterly failover drills that verify not just uptime but write correctness during the transition. Specific checks: are amounts correct? Are foreign-key relationships intact? Do aggregates match?
3. Immutable audit log - every financial transaction writes to an append-only audit log (separate from the primary database) with a checksum. Reconciliation runs hourly against this log, not monthly.

Lessons:

• Available ≠ reliable. The system can be up and serving wrong answers. Availability monitoring misses correctness failures entirely.
• Silent data corruption is more dangerous than downtime. Users know when the system is down; they don't know when the data is wrong.
• Correctness testing must cover failure scenarios (failover, network partition, concurrent writes) - not just happy-path functionality.
• Automated write verification (write → read-back → compare) catches correctness failures in seconds, not weeks.
• Financial systems must verify correctness independently of the primary data path. Audit logs and reconciliation are reliability infrastructure, not compliance overhead.

Six Takeaways from This Lesson

1. Availability = "is it responding?" Reliability = "is it responding correctly?" They're different properties. A system can be available and unreliable.

2. MTBF = time between failures. MTTR = time to recovery. Availability ≈ MTBF / (MTBF + MTTR).

3. Reducing MTTR is usually higher leverage than increasing MTBF. You can't prevent all failures; you CAN make recovery fast. Invest in monitoring, auto-failover, and runbooks.

4. Fault tolerance > fault avoidance. Design for failure, not for perfection. Reliable systems fail gracefully, recover automatically, and preserve data integrity.

5. Silent correctness failures (wrong data, lost writes, ordering violations) are more dangerous than loud availability failures (server down, error page). They're invisible until audit.

6. Automated correctness verification (write → read-back → compare, continuous reconciliation) catches what availability monitoring misses. Build it for any system where data correctness has financial or legal consequences.

Try It

Exercise: Warmup (15 minutes). Your system fails every 14 days (MTBF = 336 hours). Current MTTR is 2 hours. Calculate availability. Now: which improves availability more - doubling MTBF to 28 days, or halving MTTR to 1 hour? Calculate both and compare.

Exercise: Core (25 minutes). For a system you work with, list three ways it could be available but unreliable (serving wrong data, losing writes, returning stale results). For each, describe: how would you detect it? How long would it take to notice with current monitoring? What automated check could catch it faster?

Exercise: Stretch (45 minutes). Read the Google SRE chapter on "Embracing Risk" (link below). Note: how Google frames reliability as a feature with a cost, not an absolute requirement. How does the error budget concept connect MTBF, MTTR, and business decisions?

What's Next

Lesson 2.4 covers Single Points of Failure (SPOFs) - the components in your architecture whose failure brings down everything, and how to find and eliminate them.

Back to System Design Fundamentals