Q&A
What is fault tolerance?
Overview
In any large system, components will fail - disks die, networks partition, servers crash, dependencies time out. Fault tolerance is designing so that these failures don't cause total outages. Rather than trying to prevent every failure (impossible), fault-tolerant systems expect failure and continue functioning through redundancy and recovery mechanisms.
graph TD
A[Fault tolerance] --> B[Redundancy: no single point of failure]
A --> C[Detection: health checks, heartbeats]
A --> D[Failover: switch to a healthy replica]
A --> E[Graceful degradation: reduced service, not total failure]
A --> F[Recovery: retries, self-healing, restore]
style A fill:#1e1b4b,color:#fff
Core Mechanisms
1. Redundancy
Eliminate single points of failure by duplicating critical components - multiple servers, replicated databases, multiple availability zones/regions. If one fails, others carry the load.
2. Failure Detection
You can't react to a failure you don't notice. Health checks, heartbeats, and timeouts detect when a component is down or unresponsive.
3. Failover
When a component fails, traffic is automatically redirected to a healthy backup - promoting a database replica to primary, routing around a dead server via a load balancer.
4. Graceful Degradation
Rather than failing entirely, the system reduces functionality. If the recommendation service is down, an e-commerce site still shows products (just without personalized recommendations). Partial service beats no service.
5. Recovery and Self-Healing
Failed components are retried, restarted, or replaced automatically (e.g. orchestration restarting a crashed container, auto-scaling replacing an unhealthy instance).
Common Patterns
| Pattern | Purpose |
|---|---|
| Replication | Multiple copies of data/services for redundancy |
| Load balancing | Distribute traffic; route away from failed nodes |
| Retry with backoff | Re-attempt transient failures, spacing attempts |
| Circuit breaker | Stop calling a failing dependency to prevent cascading failure |
| Bulkhead | Isolate resources so one failure doesn't sink everything |
| Timeout | Don't wait forever on a hung dependency |
| Graceful degradation / fallback | Reduced functionality instead of total failure |
The circuit breaker is especially important: if a downstream service keeps failing, the breaker "trips" and fails fast (or returns a fallback) instead of piling up requests, which prevents one failure from cascading across the system.
Fault Tolerance vs High Availability vs Resilience
- High availability (HA) - minimizing downtime (measured in "nines": 99.9%, 99.99%).
- Fault tolerance - continuing correct operation through failures (often with no perceptible interruption).
- Resilience - the broader ability to absorb failures and recover.
Fault tolerance is one of the main ways you achieve high availability. A fully fault-tolerant system masks failures entirely; an HA system minimizes their duration.
The Reality at Scale
Failures aren't rare edge cases - at scale they're constant. This drives "design for failure" thinking: assume components will fail and build mechanisms to handle it. Concepts like chaos engineering (deliberately injecting failures, e.g. Netflix's Chaos Monkey) test whether fault tolerance actually works.
Trade-offs
Fault tolerance costs money (redundant infrastructure) and complexity (failover logic, consistency challenges during failover, testing). You apply it proportionally to how critical the system is - a payment system warrants far more than an internal dashboard.
Why This Matters in Interviews
System-design interviews lean heavily on fault tolerance. Cover the mechanisms (redundancy → no single point of failure, detection, failover, graceful degradation, recovery) and the key patterns (replication, load balancing, retries with backoff, circuit breaker, bulkhead, timeouts). Distinguish fault tolerance from high availability and resilience. The mature framing - "at scale, failure is the normal case, so design for it" - and naming circuit breakers to prevent cascading failures signal senior-level thinking.
Common Pitfalls
- Designing a single point of failure (one DB, one server) and calling it reliable.
- Adding retries without backoff/limits, amplifying load during an outage (retry storms).
- No timeouts, so a hung dependency blocks threads and cascades.
- Conflating fault tolerance with high availability (related but distinct).
- Never testing failover - untested recovery often fails when it's actually needed.
References
- Designing Data-Intensive Applications (Kleppmann), reliability chapter.
- Microsoft Azure resiliency patterns: https://learn.microsoft.com/en-us/azure/architecture/patterns/category/resiliency