Free lesson · System Design Fundamentals

Single Point of Failure (SPOF)

Easy8 min readsystem-designhlddistributed-systems

Lesson 2.4 - Single Point of Failure (SPOF)

The fourth lesson of Module 2.

A single point of failure is any component in your system whose failure brings down everything. One database with no replica. One load balancer with no backup. One data center with no failover region. One engineer who's the only person who knows how the deployment works. SPOFs are the most common reliability flaw in systems that haven't been reviewed for production readiness - and they're also the most straightforward to fix once you learn to see them. The skill isn't the fix (it's almost always "add a second one"); the skill is seeing them in an architecture diagram before they cause an outage.

Spotting SPOFs

The technique is simple: for every component in your architecture, ask one question - "if this component dies right now, does the system still work?" Walk every path from the client to the data store. Every component where the answer is "no" is a SPOF.

flowchart TD
    Users["Users (internet)"]
    LB["Load Balancer\n⚠️ SPOF if single"]
    
    S1["API Server 1"]
    S2["API Server 2"]
    S3["API Server 3"]
    
    DB_Primary["Database Primary\n⚠️ SPOF if no replica"]
    
    Cache["Redis Cache\n⚠️ SPOF if single node"]
    
    Users --> LB
    LB --> S1 & S2 & S3
    S1 & S2 & S3 --> DB_Primary
    S1 & S2 & S3 --> Cache

    style LB fill:#fb7185,stroke:#f43f5e,color:#000
    style DB_Primary fill:#fb7185,stroke:#f43f5e,color:#000
    style Cache fill:#fb7185,stroke:#f43f5e,color:#000
    style S1 fill:#34d399,stroke:#10b981,color:#000
    style S2 fill:#34d399,stroke:#10b981,color:#000
    style S3 fill:#34d399,stroke:#10b981,color:#000

An architecture diagram with three single points of failure highlighted in red. Users connect to a single load balancer - if it fails, no traffic reaches any server. The load balancer distributes to three API servers - this layer is redundant because any of the three can serve requests. But the three servers all connect to a single database primary - if it fails, no server can read or write data. The servers also connect to a single Redis cache node - if it fails, every cache lookup fails and all traffic hits the database directly, likely overwhelming it. The API server layer (three instances) is the only redundant component. The load balancer, database, and cache are all single points of failure. Any one of them failing brings the system down - either directly (load balancer, database) or indirectly through cascade (cache failure causing database overload). The irony: the team added three API servers for redundancy but forgot to add redundancy at every other layer. This is the most common SPOF pattern - engineers add redundancy where it's obvious (compute servers) and miss it where it's critical (load balancers, databases, caches, DNS).

The common SPOFs engineers miss:

1. The load balancer itself. Engineers add redundant servers behind a load balancer - three, five, ten servers. Then they have a single load balancer in front. If the LB dies, all ten servers are unreachable. The fix: a redundant LB pair (active-passive with failover, or a cloud-managed LB like AWS ALB which is inherently redundant across availability zones).

2. The database. Application servers are stateless - lose one, the others handle the traffic. The database holds all the state. Lose the database, everything stops. The fix: a replica that can be promoted to primary on failure (M28 - Replication). This is so critical that managed database services (AWS RDS, GCP Cloud SQL) offer automated failover as a standard feature.

3. DNS. Your entire infrastructure - redundant servers, replicated databases, multi-AZ deployment - is reachable via a domain name. If your DNS provider has an outage, no client can resolve your domain to an IP address. All that internal redundancy becomes irrelevant. The fix: multiple DNS providers. Configure your domain registrar to use both Route 53 and Cloudflare. If one fails, the other resolves.

4. The single availability zone. Your servers, database, and cache are all in us-east-1a. An AZ-level failure (power outage, cooling failure, network partition to the AZ) takes everything down. The fix: deploy across multiple AZs. AWS, GCP, and Azure all support multi-AZ deployment for every managed service.

5. The deployment pipeline. One CI/CD server. One deployment script. One engineer who knows how to run it. If any of these is unavailable during an incident (need to deploy a hotfix at 3am, but Jenkins is down and the one engineer who knows the manual process is on a flight), you can't deploy the fix. The fix: resilient CI/CD (managed services like GitHub Actions, redundant runners), documented deployment runbooks, and cross-training so multiple people can deploy.

6. The monitoring system. Your monitoring server (Prometheus, Datadog agent, CloudWatch collector) is itself a SPOF - if it's down, you can't detect other failures. You're flying blind. The fix: separate monitoring infrastructure from application infrastructure. The monitoring system should be independently deployed, ideally in a different failure domain (different AZ or even different account/project).

7. Shared dependencies. All 12 of your microservices use the same Redis cluster for caching. The Redis cluster is a shared dependency - if it fails, all 12 services are affected simultaneously. The fix: either make the Redis cluster itself highly available (Redis Sentinel or Redis Cluster mode), or design services to degrade gracefully when cache is unavailable (fall back to database reads - M42, Graceful Degradation).

8. Human SPOFs. One engineer who understands the billing system. One DBA who knows the database schema. One DevOps engineer who can configure the Kubernetes cluster. If that person is unavailable (sick, on vacation, quit), the team can't operate, debug, or evolve that component. The fix: documentation, cross-training, pair programming, and runbooks. The "bus factor" - how many people need to be hit by a bus before the project can't continue - should be at least 2 for every critical system.

Eliminating SPOFs: The Redundancy Pattern

The fix for every SPOF is the same principle: redundancy. Add a second instance that can take over when the first fails. The specifics vary by component:

The cost of redundancy: every redundant component costs money (two databases instead of one doubles the database cost) and adds complexity (replication lag, failover logic, split-brain prevention - all covered in later modules). Not every SPOF needs to be eliminated. The decision is a cost-benefit analysis: what's the cost of this component's failure (downtime duration × business impact per hour) versus the cost of adding redundancy (infrastructure + complexity + operational overhead)?

For a hobby project, a single database is fine - downtime costs nothing. For an e-commerce checkout, a single database is unacceptable - every minute of downtime is lost revenue. For a payment processing system, even a single-AZ deployment is unacceptable - a regional failure could violate SLAs.

SPOF Analysis in Practice

The exercise of SPOF analysis should happen at three points:

1. Design review. Before building, draw the architecture and walk every dependency. Flag every SPOF. Decide which ones to address now and which to accept as known risks.

2. Post-incident review. After every outage, ask: "was this caused by a SPOF?" If a single component failure caused a system-wide outage, the architecture has an unaddressed SPOF. Add redundancy to the post-incident action items.

3. Periodic audit. Quarterly, review the architecture for SPOFs that have crept in - a new service that depends on a single Redis instance, a new integration that goes through a single API gateway, a critical runbook that only one person has ever executed.

A Brandon Story: The DNS That Took Down Everything

A team I'd advised had built a beautifully redundant architecture: three API servers behind an ALB, a PostgreSQL primary with two read replicas and automated failover, a Redis cluster with three nodes, multi-AZ deployment across three availability zones, automated monitoring with PagerDuty integration. They were proud of their availability - 99.97% over 18 months.

Then their DNS provider had a 45-minute outage. The provider was a small, inexpensive DNS hosting company - a single point of failure that nobody had considered because it was "someone else's infrastructure." During the 45 minutes, no client anywhere in the world could resolve the company's domain name. Every server was running, every database was healthy, every monitoring check was green - but nobody could reach any of it.

The fix took 20 minutes and cost $50/month: add Cloudflare as a secondary DNS provider, configure the domain registrar to use both providers' nameservers. If one provider fails, the other resolves. The DNS SPOF was the cheapest to fix and the most impactful - because DNS is the first step for every client connection.

Three months later, their primary DNS provider had another outage. This time, Cloudflare handled all resolution. Zero user impact. The team received an alert ("primary DNS health check failed") and monitored the situation. No pages, no scramble, no outage. The $50/month investment had already paid for itself.

Lessons:

• SPOFs hide in infrastructure you don't operate yourself - DNS providers, CDNs, cloud control planes, certificate authorities
• "Someone else's problem" is still YOUR single point of failure if your system depends on it
• DNS is the most commonly overlooked SPOF because it's invisible when it works - and catastrophic when it doesn't
• The cheapest SPOFs to fix are the external ones: add a secondary provider for DNS, CDN, and monitoring
• Periodic SPOF audits should include EXTERNAL dependencies, not just internal components
• The $50/month secondary DNS was the highest-ROI reliability investment in their entire infrastructure budget

Six Takeaways from This Lesson

1. A SPOF is any component whose failure kills the system. Find them by asking "if this dies, does everything still work?" for every component.

2. The fix is always redundancy - a second instance that takes over. LB pairs, database replicas, multi-AZ deployment, multiple DNS providers.

3. Eight commonly missed SPOFs: load balancer, database, DNS, single AZ, deployment pipeline, monitoring system, shared dependencies, and human knowledge silos.

4. Not every SPOF must be eliminated. The decision is cost-of-failure vs cost-of-redundancy. Hobby projects accept SPOFs; payment systems don't.

5. External dependencies are SPOFs too - DNS providers, CDNs, cloud control planes. "Someone else's infrastructure" is still your dependency.

6. Human SPOFs are real. One engineer who knows the system = a bus factor of 1. Fix with documentation, cross-training, and runbooks.

Try It

Exercise: Warmup (15 minutes). Draw the architecture of a system you work with (or a system you use - Instagram, Uber, your company's app). Walk every component and flag SPOFs. How many did you find? Which is the highest-risk?

Exercise: Core (25 minutes). For each SPOF you identified, design the redundancy fix: what component do you add? What's the failover mechanism? What's the approximate cost? Rank them by risk-reduction-per-dollar.

Exercise: Stretch (30 minutes). Audit your external dependencies: DNS provider, CDN, monitoring service, CI/CD platform, email delivery, payment processor. For each, answer: if this provider has a 1-hour outage, what's the impact on your system? Do you have a secondary provider or degradation strategy?

What's Next

Lesson 2.5 covers the three performance numbers - latency, throughput, and bandwidth - and why engineers constantly confuse them.

Back to System Design Fundamentals