AlgoInsight

System Design

Single Points of Failure: The Outage That Taught Me to Hunt Them

Everything was redundant. Three of everything. Except one small component nobody thought about - and when it died, it took the whole system with it.

By Admin2026-06-1810 min readsingle-point-of-failurereliabilityhigh-availabilitydistributed-systemsredundancysystem-design

I was on call the night a service I owned went completely dark. The frustrating part wasn't that it failed - things fail. It was that we'd been careful. Three application servers. A replicated database. Load balancers in pairs. We'd done the redundancy homework.

And then a single small component - one config service that every request quietly depended on to start up - fell over. It had no backup. Nobody had drawn it on the architecture diagram as a risk because it felt too boring to matter. When it died, all three of our beautifully-redundant app servers couldn't function, and the whole system went down behind one unglamorous box.

That night taught me a habit I've carried into every design review since: hunt single points of failure before they hunt you. Here's how to do it.

What a single point of failure is

A single point of failure (SPOF) is any component whose failure, on its own, brings down the entire system. If you can point at one box, one service, one database, one anything - and removing it stops everything - that's a SPOF.

The classic suspects are easy to name:

The dangerous ones are the suspects you don't name - like my config service. The shared cache everything reads from. The DNS that has to resolve before anyone connects. The single message broker the whole pipeline flows through. The deploy system that, if broken, means you can't even ship the fix. SPOFs love to hide inside the things you've stopped noticing.

graph LR
    subgraph BAD[Has a SPOF]
        C1[Clients] --> LB1[Single Load Balancer]
        LB1 --> A1[App] & A2[App]
        A1 & A2 --> DB1[(Single Database)]
    end
    subgraph GOOD[No SPOF]
        C2[Clients] --> LB2[LB] & LB3[LB standby]
        LB2 --> A3[App] & A4[App]
        A3 & A4 --> DBP[(DB Primary)]
        DBP -.replicates.-> DBR[(DB Replica)]
    end

This diagram compares two architectures. The first, labeled "has a SPOF," shows clients going through a single load balancer to two app servers, which both depend on a single database. The single load balancer and single database are each single points of failure. The second, labeled "no SPOF," shows clients reaching a primary load balancer with a standby, app servers, and a database primary that replicates to a replica - so no single component failing can take the whole system down.

How to actually find them

The technique is almost embarrassingly simple, and almost nobody does it rigorously. Trace one request's entire path, and at every single hop ask: "what happens if this one thing dies right now?"

Client → DNS → load balancer → app server → cache → database → and back. Walk every arrow. If the honest answer at any hop is "everything stops," you've found a SPOF. Then ask the harder question: what about the things that aren't on the happy path? The config service. The secrets store. The service-discovery registry. The certificate authority. These don't appear in the obvious request flow, which is exactly why they bite.

Tip: Run this as a literal exercise on your own design before an interview: name every component, then cross out each one in turn and ask "does the system survive?" The components whose removal kills everything are your SPOFs. This is the same walkthrough I do in every architecture review I sit in.

The math that makes this concrete

Here's the part that surprised me when I first ran the numbers, and the part that makes a great interview point. Availability multiplies along a chain. If a request must pass through several components in series, the overall availability is the product of their individual availabilities. One weak link drags the whole chain down to its level - no matter how good everything else is.

def serial(components):                 # chain: every component must be up
    a = 1.0
    for c in components:
        a *= c
    return a

def redundant(component, n):            # n parallel copies: only one must be up
    return 1 - (1 - component) ** n

# A request path: three "four nines" components and one "two nines" SPOF
chain = [0.9999, 0.9999, 0.99, 0.9999]   # the 0.99 is the weak link
a = serial(chain)
print(f"With the SPOF:        {a*100:.3f}%   (~{(1-a)*8760:.0f} hours down/year)")

# Replace that one SPOF with 3 redundant copies of itself
chain_fixed = [0.9999, 0.9999, redundant(0.99, 3), 0.9999]
b = serial(chain_fixed)
print(f"After adding redundancy: {b*100:.4f}% (~{(1-b)*8760:.1f} hours down/year)")

Run it. Three of your four components are "four nines" (99.99%) - excellent. But that one 99% component drags the entire system down to roughly 99%, which is about 90 hours of downtime a year. Your beautiful four-nines components are wasted; you're only as available as your weakest link.

Now watch what redundancy does. Replace that single weak component with three copies of itself, where the request only needs one of them alive, and the system jumps to four-nines - a couple of hours of downtime a year. The entire improvement came from eliminating one SPOF. That's the leverage. That's why hunting them is the highest-return reliability work you can do.

Interview: In a system design loop, after you sketch your architecture, proactively say: "Let me check for single points of failure." Then walk your own diagram and point out where you'd add redundancy. I cannot overstate how much this impresses an interviewer - it shows you think about failure, not just the happy path. The candidates who do this unprompted are the ones I write "strong hire" for.

Redundancy isn't free (the senior caveat)

If the answer were simply "make three of everything," this wouldn't be an interesting topic. Removing SPOFs introduces its own costs, and naming them is what separates a thoughtful design from a reflexive one:

That last one is the subtle, senior-level point. Counting copies isn't enough; you have to ask whether those copies fail together. My config-service outage was, in a sense, a correlated-failure lesson - the redundant app servers shared a single hidden dependency, so they were never as independent as the diagram suggested.

The shift that night gave me wasn't "add more redundancy." It was a discipline: assume every component will fail, trace the consequences before reality does it for you, and spend your redundancy budget on the links that actually cap your availability. That mindset - designing for failure as the default, not the afterthought - is woven through all of System Design Fundamentals. The happy path is the easy part. The interview, and the 3 a.m. page, are about everything else.

Go deeper

More articles