System Design

How to Handle Failures in Distributed Systems

At scale, the question is never whether something is broken right now. It's how many things - and whether your system noticed, or just fell over with it.

By Admin2026-06-1810 min readdistributed-systemsreliabilityfailure-handlingcircuit-breakerresiliencesystem-design

When I moved from building systems on a handful of servers to working on systems spread across thousands of machines, one idea reorganized how I think about everything: at large scale, something is always failing. A disk somewhere, a network link, a process that just got OOM-killed, a dependency having a bad minute. It's not an exception to plan around; it's the steady state.

That reframing flips the goal. You don't design a distributed system hoping nothing breaks. You design it assuming things constantly break, and you make sure individual failures stay small instead of cascading into an outage. The companion to this is detecting failure - which I cover separately in the piece on heartbeats - but this post is about the harder half: what to do once a part of your system goes down.

The mindset: contain, don't prevent

You cannot prevent failure in a distributed system. You can only control how far it spreads. Almost every resilience technique below is really the same idea wearing different clothes: stop one component's failure from becoming everyone's failure.

Let me walk the toolkit, roughly in the order you'd reach for each.

1. Timeouts - never wait forever

The most basic failure-handling tool, and the one people forget. If your service calls another service, and that service hangs, what happens? Without a timeout, your request waits. And waits. The thread is stuck, holding resources. Soon every thread is stuck waiting on the same dead dependency, and now your service is down too - killed by something downstream.

A timeout says "if I don't hear back in N milliseconds, I give up and move on." It's the difference between one slow dependency causing a blip versus taking your whole service with it. Every network call needs one. No exceptions.

Warning: A surprising number of outages trace back to a missing timeout. One hung dependency, no timeout, and the stall propagates upward through every service waiting on it. It's the most boring resilience tool and the one most often skipped.

2. Retries - but carefully

If a request fails because of a transient blip - a brief network hiccup - retrying often just works. But naive retries are dangerous in two specific ways.

First, retries need idempotency. If the operation isn't safe to repeat (like charging a card), a retry can do real damage. (I wrote a whole piece on why idempotency is non-negotiable for exactly this reason.)

Second, retries can amplify an outage. If a service is struggling and every client immediately retries, you've just multiplied the load on an already-failing service - a "retry storm" that turns a small problem into a total collapse. The fix is exponential backoff with jitter: wait a little longer between each retry (1s, 2s, 4s…), and add a small random delay so a thousand clients don't all retry at the exact same instant. Backoff gives the struggling service room to recover; jitter spreads the load out.

3. Circuit breakers - stop hammering the dead

This one is my favorite because it's so intuitive once you see it. If a dependency is clearly down - failing every call - what's the point of continuing to call it? You're just wasting time, piling on retries, and slowing your own responses waiting for timeouts.

A circuit breaker wraps a dependency and watches its failures. Borrowed straight from electrical engineering: when failures cross a threshold, the breaker "trips" and stops calling the dependency entirely for a while - failing fast instead. After a cooldown, it cautiously lets a trial request or two through; if they succeed, it closes again and resumes normal traffic.

stateDiagram-v2
    [*] --> Closed
    Closed --> Closed: calls succeed
    Closed --> Open: failures exceed threshold
    Open --> Open: fail fast (skip the call)
    Open --> HalfOpen: cooldown timer elapses
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails

This state diagram shows a circuit breaker's three states. It starts Closed, where calls flow normally and succeed. If failures exceed a threshold, it moves to Open, where it fails fast and skips calling the dependency entirely. After a cooldown timer elapses, it moves to Half-Open and allows a trial call. If that trial succeeds, it returns to Closed; if it fails, it goes back to Open.

class CircuitBreaker:
    def __init__(self, threshold=3):
        self.threshold = threshold
        self.failures = 0
        self.state = "CLOSED"

    def call(self, dependency_ok):
        if self.state == "OPEN":
            return "fail-fast (skipped - breaker open)"
        if dependency_ok:
            self.failures = 0
            return "success"
        self.failures += 1
        if self.failures >= self.threshold:
            self.state = "OPEN"
            return "FAILURE - breaker TRIPPED open"
        return "FAILURE"

cb = CircuitBreaker(threshold=3)
for ok in [True, False, False, False, True, True]:
    print(f"dependency_ok={str(ok):5} -> {cb.call(ok):35} [state={cb.state}]")

Run it and watch the key moment: after three failures the breaker trips to OPEN, and then the next calls are skipped even though the dependency recovered (ok=True). That's the protection - the breaker shields you from a dependency you believe is dead, and only re-checks after a deliberate cooldown rather than hammering it on every request.

4. Graceful degradation - serve something

When a component fails, can you give the user a worse but still useful answer instead of an error? If the recommendation service is down, show generic popular items instead of a blank page. If live inventory is unavailable, show the last-known cached value with a soft "may be out of date." A degraded experience usually beats a broken one. The senior instinct is to design fallbacks before the failure, so there's always something to serve.

5. Bulkheads and load shedding - protect the whole from the part

Two more containment ideas worth naming:

• Bulkheads (named after a ship's watertight compartments): isolate resources so one overloaded feature can't drain the pool every other feature needs. Give the flaky dependency its own connection pool, so when it floods, it floods its own compartment and the ship stays afloat.
• Load shedding: when you're genuinely overwhelmed, deliberately reject some requests so you can serve the rest. Returning a fast "try again" to 10% of traffic is far better than collapsing for 100% of it. Surviving degraded beats dying complete.

Putting it together

Interview: When an interviewer asks "what happens if this service goes down?", they're handing you a gift. Don't just say "it'll fail." Walk the toolkit: "I'd put a timeout on the call, wrap it in a circuit breaker so we fail fast if it's down, and degrade gracefully by serving cached data." That sequence signals you design for the real world, where things break constantly - not just the happy path.

The shift from "prevent failure" to "contain failure" is, to me, the line between thinking about systems and thinking about distributed systems. Detection comes first (that's heartbeats), but containment is what keeps a 3 a.m. page from becoming a 3 a.m. outage. We build out this whole resilience toolkit, with the trade-offs of each, in System Design Fundamentals.

Go deeper

• Circuit breaker design pattern - the pattern, its states, and where to apply it.
• Fault tolerance - the broader discipline of designing systems that survive component failure.
• Google SRE books - free, canonical reading on handling failure and overload in production.
• Amazon Builders' Library - real-world patterns for timeouts, retries, backoff, and graceful degradation.

More articles