Free lesson · System Design Crash Course

Failure as a Design Dimension

Easy12 min readsystem-designhlddistributed-systems

Lesson 1.5 - Failure as a Design Dimension

The mindset-shift lesson.

There's a specific moment in many engineers' careers when their thinking flips on a fundamental question. Up until that moment, when they design something, they're thinking about how it should work. After that moment, they're thinking about what happens when it doesn't.

This shift is one of the biggest differences between a junior engineer and a senior one. And it's not because senior engineers are pessimists. It's because, after enough years of operating real systems, they've learned that everything fails, eventually, in some way you didn't anticipate. Not in a dramatic way. Just regularly, quietly, often. Servers crash. Networks drop packets. Disks corrupt data. Code has bugs. People make mistakes. Cloud providers have outages. The internet itself has bad days.

If you design a system assuming all of that doesn't happen, you build something that works on your laptop but breaks in production. If you design a system assuming all of that does happen - and plan for it - you build something that survives the real world.

This lesson is about making that mindset shift early in your career, instead of after years of painful incidents. Welcome to the way senior engineers think about systems.

Everything Fails

Let me give you an honest list of things that fail in production, from the most common to the most surprising:

Software bugs. Every codebase has them. The code you ship today has bugs you don't know about yet. Bugs are guaranteed.

Server crashes. Individual servers crash. They might run for a year, then crash. Out of memory, kernel panic, hardware fault, software hang. Every fleet of servers has some percentage that's misbehaving at any given moment.

Network issues. Packets get dropped, delayed, reordered. Sometimes a network link just goes down. Sometimes DNS records get corrupted. Sometimes traffic routes through unexpected paths and adds latency.

Disk failures. Hard drives fail with surprising regularity. At Google's scale, a few drives fail every day. Even SSDs fail eventually.

Database issues. Databases run out of space. They fall behind on replication. They get into weird states that require manual recovery. They have query plans that go bad after a schema change.

Configuration mistakes. Someone deploys a config change that breaks production. Someone removes a server they thought was unused. Someone misconfigures a firewall rule.

Third-party outages. Your service depends on Stripe, AWS, Twilio, SendGrid. When they have outages, you have outages. They have outages more often than you'd think.

Code deployments. Most production incidents come from recent changes. Even with good testing, some bad changes make it through.

Capacity issues. Traffic spikes overwhelm your system. A celebrity tweets about you and you go down.

Human errors. "I accidentally dropped the production database." This is more common than companies admit. (See: GitLab's famous 2017 incident.)

Cosmic rays. Yes, really. At Google's scale, cosmic radiation flips memory bits often enough to cause noticeable issues. Hence ECC memory.

The thing you didn't expect. Every postmortem ever written has some flavor of "we did not anticipate that X could fail this way." The space of possible failures is bigger than your imagination.

This list isn't fearmongering. It's just the reality of operating systems. If you can accept this list as true, you're in the right mindset for what comes next.

The Mental Shift

Here's the shift you need to make. When you design a system, don't just ask:

"How does this work?"

Also ask:

"What happens when this fails?"

For every component in your design. For every connection between components. For every external dependency. The discipline is to systematically consider failure for every box and arrow in your diagram.

This isn't pessimism. It's engineering. The same way a bridge engineer doesn't just design for normal traffic - they also design for high winds, earthquakes, and overloads - software engineers design for normal operation AND failure modes.

A simple way to start: take any HLD diagram and put a sticky note on each component. On the note, write: "What if this fails?" Now you have a list of failure scenarios to think through.

Let me show you what this looks like with our familiar HLD from Lesson 1.3:

flowchart LR
    User["User's browser<br/>❓ slow network"]
    LB["Load Balancer<br/>❓ LB crashes"]
    App["App Servers<br/>❓ Server crashes<br/>❓ Code bug"]
    Cache["Cache<br/>❓ Cache evicts<br/>❓ Cache down"]
    DB["Database<br/>❓ DB slow<br/>❓ DB down<br/>❓ Disk full"]

    User --> LB
    LB --> App
    App --> Cache
    App --> DB

    style User fill:#34d399,stroke:#10b981,color:#000
    style LB fill:#fbbf24,stroke:#f59e0b,color:#000
    style App fill:#a78bfa,stroke:#8b5cf6,color:#000
    style Cache fill:#fbbf24,stroke:#f59e0b,color:#000
    style DB fill:#fb7185,stroke:#f43f5e,color:#000

A diagram of our simple web architecture, but with failure modes annotated on each component. The user's browser might have a slow network. The load balancer might crash. The app servers might crash or have code bugs. The cache might evict data unexpectedly, or the cache server might go down entirely. The database might be slow, might be down, might have a disk full. Every single component has multiple failure modes. The exercise of doing this annotation  - going through each box and asking "what can go wrong here?"  - is the start of designing for failure. Once you have the list, you can then design responses to each failure mode.

This is exactly the activity senior engineers do. Take your design, annotate it with failure modes, then design responses to each one.

The Core Failure-Handling Patterns

You don't need to invent failure handling - there are well-known patterns. The five most important for beginners to learn:

Pattern 1: Eliminate single points of failure

A single point of failure (SPOF) is any component that, if it fails, takes the whole system down. The classic example: if you have one database, and the database goes down, your whole app is down. The database is a SPOF.

The fix: redundancy. Have more than one of every critical component. Two app servers instead of one. A database with a hot standby. Two load balancers (with the DNS pointing to both).

flowchart LR
    subgraph SPOF["⚠️ With SPOF"]
        App1A["App Server"]
        DB1A["Database"]
        App1A --> DB1A
    end

    subgraph Redundant["✓ Without SPOF"]
        App2A["App Server 1"]
        App2B["App Server 2"]
        DB2A["Primary DB"]
        DB2B["Standby DB"]
        App2A --> DB2A
        App2B --> DB2A
        DB2A -.->|"replicate"| DB2B
    end

    style SPOF fill:#fb7185,stroke:#f43f5e,color:#000
    style Redundant fill:#34d399,stroke:#10b981,color:#000

Two architectures compared. On the left, a system with a single point of failure  - one app server connected to one database. If either goes down, the system is down. On the right, the same system but with redundancy. Two app servers handle traffic, both connected to a primary database. A standby database replicates from the primary. If the primary database fails, the standby can take over. If one app server fails, the other handles all the traffic. Redundancy at every layer eliminates single points of failure. This is the foundation of reliability  - you don't try to prevent any single component from ever failing, you make sure no single failure takes down the whole system.

Spotting SPOFs in a design is the first systematic failure analysis. Scan your diagram. If any box's failure would take everything down, you have a SPOF. Add redundancy or accept the risk.

Interactive animation: When the active node dies, a warm standby takes over and traffic is redirected to it automatically.

Pattern 2: Timeouts on everything

When your code calls another service, that call might never return. The other service might be hung, the network might have dropped the response, the universe might have ended. If you wait forever, your service will lock up.

The fix: timeouts. Every call to anything that crosses a process or machine boundary should have a timeout. If the response doesn't come within (say) 5 seconds, give up.

# WRONG: no timeout, can wait forever
response = http_client.get('https://other-service/api/data')

# RIGHT: explicit timeout
response = http_client.get('https://other-service/api/data', timeout=5)

Junior engineers often forget timeouts. Senior engineers add them by default. The default in many libraries is "no timeout," which is a terrible default.

A practical rule: assume any external call will sometimes hang. Every HTTP call, every database query, every cache lookup, every queue operation - all of them need timeouts.

Pattern 3: Retries with backoff

Sometimes failures are transient - a brief network blip, a quick server hiccup. Retrying often works. But blind retrying is dangerous: if the other service is overloaded, retrying makes it worse.

The standard pattern is retry with exponential backoff and jitter.

Exponential backoff: wait longer between each retry. First retry after 1s, then 2s, then 4s, then 8s. Don't pound the failing service.
Jitter: add randomness to the wait times so that all your clients aren't retrying at exactly the same moment.
A retry budget: don't retry forever. Give up after some number of attempts.

# Pseudocode
for attempt in 1..5:
    try:
        return call_service()
    except TransientError:
        wait_time = (2 ** attempt) + random.uniform(0, 1)  # exponential + jitter
        sleep(wait_time)
raise ServiceFailedError()

A common beginner mistake: retry without backoff or jitter. If a service is overloaded and 1000 clients all retry immediately, the service gets even more overloaded. The pattern of "retry with backoff and jitter" prevents this.

Interactive animation: A client calls a service under a retry policy: a timeout on each attempt, then backoff with jitter between retries.

Pattern 4: Graceful degradation

When something fails, the ideal is "the whole system keeps working" - but sometimes that's impossible. The next best is "the system keeps working in reduced form."

Examples:

The recommendations service is down. The product page still loads, but without recommendations.
The cache is down. The app is slower but still functional (queries hit the database directly).
The email service is down. Users see a "we'll email you the receipt soon" message, and the system queues the work for later.

This is graceful degradation. When something fails, you fall back to a less-good but still-working experience.

Compare to catastrophic failure: the recommendations service is down, so the entire product page errors out and shows "500 Internal Server Error" to the user. Bad design.

The discipline is to ask, for every component: "If this fails, can the system continue in some reduced form?" Often the answer is yes, with a small code change. That's a big resilience win for a small effort.

Interactive animation: Graceful versus catastrophic: with recommendations down, the page still loads without them instead of showing a 500 error.

Pattern 5: Circuit breakers

When a downstream service is failing, you don't want to keep calling it and waiting for timeouts. Eventually, those timeouts pile up and degrade your own service.

A circuit breaker is a pattern where, after enough failures, you "open the circuit" and stop calling the failing service for a while. You fail fast (return an error immediately) instead of waiting for the timeout. After some time, you "half-open" the circuit and try one call to see if the service recovered.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: Many failures
    Open --> HalfOpen: After timeout
    HalfOpen --> Closed: Success
    HalfOpen --> Open: Failure

A state diagram showing the circuit breaker pattern. The breaker starts in the closed state, meaning calls flow through normally. When failures accumulate past a threshold, the breaker transitions to open  - calls fail fast without even being attempted. After a timeout period, the breaker transitions to half-open  - it lets one or a few calls through to test if the downstream service has recovered. If those test calls succeed, the breaker goes back to closed and normal traffic flow resumes. If they fail, it goes back to open and waits longer before trying again. This pattern protects your own service from being dragged down by a failing dependency, while still automatically recovering when the dependency comes back.

Circuit breakers prevent cascading failures - one dependency failing doesn't drag down the whole system. They're a standard pattern in service-to-service communication, especially in microservice architectures.

There are five more patterns you'll learn over time (bulkheads, fallbacks, dead-letter queues, idempotency, distributed transactions), but the five above are the starting toolkit. Internalize them and you're already designing more resiliently than most beginners.

Interactive animation: After enough failures the breaker opens and calls fail fast, then a half-open trial call decides whether it closes.

A Real Failure Example

Interactive animation: A cascading failure: a 30-second database hiccup becomes more than ten minutes of user-visible errors.

Let me give you a concrete story. (This is a composite of real incidents, simplified for clarity.)

A company runs a web app. Architecture: load balancer → app servers → database. They've been running fine for months. One Tuesday at 3 PM, the database server has a brief network hiccup. It's unreachable for 30 seconds, then recovers.

What happens? Let me walk through the failure cascade:

The 100 app servers are each handling requests. Each request needs the database.
Database becomes unreachable. Each app server has a 30-second timeout on database calls.
All currently-running requests block, waiting for the database.
New requests keep arriving. The app servers accept them, then they also block.
Within 10 seconds, every app server has hit its concurrent-request limit. New requests get queued.
The load balancer's health checks start failing because app servers aren't responding fast enough.
The load balancer pulls "unhealthy" app servers out of rotation. Traffic concentrates on fewer servers.
Those servers also get overwhelmed.
All servers are now marked unhealthy. The load balancer returns errors to users.
The database recovers after 30 seconds.
App servers eventually recover too, but it takes minutes for the queue to drain.

Total impact: 30-second database hiccup → 10+ minutes of user-visible errors.

This is a real pattern - a "cascading failure" or "metastable failure." It happens because the system wasn't designed for the database briefly being unreachable.

What would have prevented this? A combination of patterns:

Shorter timeouts: if the database isn't responding in 1 second, don't wait 30. Fail faster.
Circuit breaker: after a few failures, stop calling the database briefly.
Graceful degradation: when the database is unreachable, serve cached data or a "service degraded" message instead of just hanging.
Bulkheading: isolate different types of requests so one type can't consume all the app server capacity.
Backpressure: when overloaded, shed load (drop some requests) rather than letting the queue grow infinitely.

These patterns aren't theoretical. They're what teams add after experiencing exactly this kind of cascading failure. The lesson is: design for failure FROM THE START, not after your first major incident.

The Honest Reality of Failure

Let me be honest with you about something. Even with all the patterns above, failures still happen. Designing for failure doesn't mean eliminating failure - it means containing failure when it occurs.

A few honest truths:

1. You can't prevent all failures. No matter how careful you are, something will surprise you. The goal isn't perfection; the goal is resilience.

2. Most failures are recoverable. Most production incidents last minutes to hours, not days. Good design means the system recovers, often without manual intervention.

3. Practice helps. Some companies (Netflix, famously) deliberately inject failures into production to test their resilience. This is called "chaos engineering." It surfaces unhandled failure modes before they bite you.

4. Postmortems are gold. When something does go wrong, the blameless postmortem afterward is one of the most valuable learning artifacts in software. Every postmortem ever written has lessons you can absorb.

5. The graph of failures gets worse at scale. A system with 10 servers fails less interestingly than a system with 10,000 servers. As scale grows, weird failure modes become routine. This is why hyperscalers (Google, Amazon, Meta) have invested so heavily in failure handling.

A Mental Habit to Build

Make this a permanent habit: every time you design something, mentally run a failure scenario. What if this server dies? What if this network call hangs? What if this dependency is down? What if the disk is full?

You don't need to write code for every failure mode. But thinking through them often reveals issues you wouldn't have spotted otherwise. A senior engineer's quiet superpower is asking, mid-design-review: "What happens if X fails?" The answer often surfaces a gap.

This habit is also tested in interviews. When you're walking through a system design and the interviewer asks "what if the load balancer fails?" - they're checking whether you've internalized failure as a design dimension. Candidates who can rattle off "we'd have a backup load balancer, here's how failover works, here's how DNS handles it" pass with strong signal. Candidates who blink and say "I didn't think about that" don't.

Six Takeaways from This Lesson

Everything fails. Servers crash, networks drop packets, code has bugs, people make mistakes. Designing as if failures don't happen produces brittle systems.
Failure is a design dimension. When you design a system, systematically ask "what happens when each part fails?" for every component.
The major failure-handling patterns are timeouts, retries with backoff, graceful degradation, circuit breakers, and eliminating single points of failure. Internalize all five.
Cascading failures are the most painful kind. A small problem amplifies into a large one because the system wasn't designed to contain it. The patterns above prevent this.
Failures still happen even with good design. The goal is resilience, not elimination. The system should recover, often automatically.
Make "what fails?" a permanent question. Senior engineers ask it about every component. Train yourself to do the same.

Try It

Exercise: Warmup (10 minutes). Take a simple architecture (your own work, or the one from Lesson 1.3). For each component, write down at least one specific way it could fail.

Exercise: Core (30 minutes). Read a published postmortem from a real company. Good sources: GitHub's incident history, the Atlassian incident database, Cloudflare's blog has many detailed postmortems. Note what went wrong, what amplified it, and what they did differently after.

Exercise: Stretch (60 minutes). Take the cascading failure example earlier in this lesson. Sketch how each of the five patterns (timeouts, retries+backoff, graceful degradation, circuit breakers, redundancy) would have changed the outcome. Which would have helped most? Which together?

What's Next

Lesson 1.6 covers how HLD evolves over a system's lifetime. The architecture you'd build on Day 1 is very different from what you'd have in Year 5, and knowing the patterns of evolution is part of being a good designer.

Back to System Design Crash Course