Free lesson · System Design Fundamentals

Challenges of Distribution

Medium11 min readsystem-designhlddistributed-systems

Lesson 19.1 - Challenges of Distribution

The first lesson of Module 19.

On a single machine, a function call takes nanoseconds and never fails (unless you're out of memory or the process crashes - and then EVERYTHING fails, not just one call). The clock is one clock. Memory is shared. Disk access is local. There are no partial failures - either the machine is up and everything works, or it's down and nothing works. Debugging is straightforward: attach a debugger, set a breakpoint, inspect memory.

Distributed systems break every one of these assumptions. A call to another machine takes milliseconds (a million times slower than a local function call) and CAN fail - independently of everything else. The calling machine might be fine while the called machine is down. Clocks on different machines drift apart. There's no shared memory - data must be copied over the network. Failures are partial: Machine A is up, Machine B is down, Machine C can reach A but not B. Debugging requires correlating logs from multiple machines that don't share a clock. Every interaction between machines involves the network, and the network is the single largest source of failure in modern infrastructure.

This lesson covers the fundamental challenges that make distributed systems categorically harder than single-machine systems - anchored on the Eight Fallacies of Distributed Computing, a set of assumptions that engineers carry from single-machine thinking and that break catastrophically in distributed environments.

The Eight Fallacies of Distributed Computing

In 1994, Peter Deutsch (at Sun Microsystems) identified seven assumptions that engineers incorrectly make about distributed systems. James Gosling later added an eighth. Thirty years later, every one of these fallacies still causes production outages.

flowchart TD
    F1["1. The network is reliable"]
    F2["2. Latency is zero"]
    F3["3. Bandwidth is infinite"]
    F4["4. The network is secure"]
    F5["5. Topology doesn't change"]
    F6["6. There is one administrator"]
    F7["7. Transport cost is zero"]
    F8["8. The network is homogeneous"]
    
    F1 --> Impact1["Packets get dropped.\nConnections time out.\nAPI calls fail silently.\nRetries are mandatory."]
    F2 --> Impact2["Cross-service calls add\n1-100ms per hop.\n5 sequential service calls\n= 5-500ms added latency."]
    F3 --> Impact3["Large payloads congest\nthe network.\nFan-out amplifies traffic.\nSerialization costs are real."]

    style F1 fill:#fb7185,stroke:#f43f5e,color:#000
    style F2 fill:#fbbf24,stroke:#f59e0b,color:#000
    style F3 fill:#a78bfa,stroke:#8b5cf6,color:#000

A diagram showing the first three of the eight fallacies of distributed computing and their real-world impact. Fallacy 1: the network is reliable. In reality, packets get dropped, TCP connections time out, API calls fail silently (the response never arrives), and entire network links go down. Any call to another machine can fail - retries, timeouts, and fallback logic are mandatory, not optional. Fallacy 2: latency is zero. A local function call takes nanoseconds. A call to another service in the same datacenter takes 0.5-5 milliseconds. A call across the internet takes 20-200 milliseconds. If your request path involves 5 sequential service calls, each adding 2ms, that's 10ms of added latency - invisible for one request but significant at scale. If one service is across the ocean, a single call adds 100-200ms. Latency is always nonzero, always variable, and compounds across service calls. Fallacy 3: bandwidth is infinite. A microservice returning a 2MB JSON payload to 1,000 clients per second generates 2 GB/s of network traffic - enough to saturate a 10 Gbps link. Fan-out patterns (one request triggers requests to 50 backend services) amplify traffic. Serialization and deserialization of large payloads consume CPU. Network bandwidth is finite, costs money, and becomes a bottleneck at scale.

Fallacy 1: The Network Is Reliable

Reality: networks fail constantly. Switches reboot. Cables get unplugged (accidentally by a datacenter technician walking through a rack). BGP routes change. DNS resolves to the wrong IP. Cloud provider network infrastructure has transient glitches that last 100ms-10 seconds - long enough to time out TCP connections but short enough that monitoring dashboards show nothing.

Implication: every network call in your code can fail. http.get("http://user-service/users/42") might return a response, might time out, might return a connection refused error, or might return a partial response (the connection dropped mid-transfer). Your code must handle ALL of these cases - not just the happy path. This is fundamentally different from userService.getUser(42) as a local function call, which either returns a value or throws a programmatic exception.

Fallacy 2: Latency Is Zero

Interactive animation: The same call at three distances: about 1 ns in-process, 0.5-5 ms in the datacenter, 80-120 ms across regions.

Reality: a function call within the same process takes ~1 nanosecond. An HTTP call to a service in the same datacenter takes 0.5-5 milliseconds - 5,000-5,000,000 times slower. A call across regions (US East to EU West) takes 80-120 milliseconds.

Implication: service call chains accumulate latency. A request that passes through Service A → B → C → D → E, with each hop taking 3ms, adds 15ms of network latency alone - before any of the services do actual work. The N+1 query problem (M15.3) is bad on a single database; the N+1 service call problem is catastrophic in microservices. A loop that makes 100 sequential calls to another service takes 100 × 3ms = 300ms of pure network latency.

Fallacy 3: Bandwidth Is Infinite

Reality: a 10 Gbps network link handles ~1.2 GB/s. A service returning 500 KB JSON responses at 5,000 requests/second generates 2.5 GB/s - exceeding the link capacity. Add serialization overhead (JSON is verbose), and the actual payload throughput is lower.

Implication: payload size matters. Sending the entire user profile (50 columns, 20 KB) when the caller only needs the name (50 bytes) wastes 400x the bandwidth. At 10,000 requests/second, that's 200 MB/s of wasted network traffic. APIs should return only the fields the caller needs (GraphQL's strength, REST's weakness unless field selection is implemented).

Fallacy 4: The Network Is Secure

Reality: traffic between services can be intercepted (man-in-the-middle on the network), spoofed (a compromised machine impersonates a service), or tampered with (a malicious actor modifies packets in transit). Even within a "private" datacenter network, lateral movement (an attacker compromises one machine and uses it to attack others on the same network) is a real and common attack pattern.

Implication: encrypt traffic between services (mTLS - mutual TLS). Authenticate service-to-service calls (service mesh, JWT tokens between services). Don't trust the network just because it's "internal."

Fallacy 5: Topology Doesn't Change

Reality: machines are added and removed constantly (auto-scaling, deployments, hardware replacement). IP addresses change (cloud instances get new IPs on restart). DNS entries are updated. Load balancers add and remove backends. In Kubernetes, pods are created and destroyed every few seconds.

Implication: don't hardcode IP addresses or hostnames. Use service discovery (DNS-based, consul, Kubernetes Services) to find other services dynamically. Assume the machine you talked to 5 minutes ago might not exist anymore.

Fallacy 6: There Is One Administrator

Reality: in a microservices architecture, different teams own different services. Team A deploys their service on Monday; Team B deploys theirs on Thursday. Team A changes their API response format; Team B's client breaks. Nobody coordinated. In multi-cloud or hybrid environments, different infrastructure teams manage different networks, firewalls, and DNS zones.

Implication: services must be independently deployable with backward-compatible APIs. API versioning (M5.5), contract testing, and change communication are operational necessities, not luxuries.

Fallacy 7: Transport Cost Is Zero

Reality: serialization (converting objects to bytes for network transfer) and deserialization (converting bytes back to objects) consume CPU. JSON serialization of a complex object: ~1ms. Protobuf serialization: ~0.1ms. At 50,000 requests/second, JSON serialization alone consumes 50 seconds of CPU per second - more than one core. Network infrastructure (load balancers, firewalls, NAT gateways) has per-request processing costs and dollar costs (AWS NAT Gateway charges per GB processed).

Implication: choose efficient serialization formats (Protobuf, Avro, MessagePack) for high-throughput internal communication. Reserve JSON for external APIs where human readability matters. Account for NAT Gateway and load balancer data processing costs in infrastructure budgets - they surprise teams at scale.

Fallacy 8: The Network Is Homogeneous

Reality: services run on different OS versions, different hardware (ARM vs x86), different container runtimes, different network stacks. One service uses HTTP/2; another speaks gRPC. A legacy service uses SOAP. The cloud provider's managed services use proprietary protocols behind SDK wrappers.

Implication: standardize internal communication protocols (gRPC for service-to-service, HTTP/REST for external APIs). Use API gateways to translate between protocols at the boundary. Don't assume all services speak the same language - build adapters.

Interactive animation: All eight fallacies, one by one: the assumptions engineers carry over from one machine, and what each one breaks.

The Three Fundamental Problems of Distributed Systems

Beyond the fallacies, distributed systems have three problems that don't exist on single machines:

Interactive animation: Partial failure, no global clock, no shared memory: the three problems that do not exist on a single machine.

1. Partial Failure

On a single machine: either everything works (machine is up) or nothing works (machine is down). In a distributed system: Machine A is up, Machine B is down, Machine C can reach A but not B. The system is partially working. Some users are served; others get errors. Some writes succeed; others fail. The system is in an inconsistent state - not fully up, not fully down.

Why this is hard: the system must continue operating despite partial failure. It can't just crash and restart like a single-machine application. It must detect which parts are failed, route around them, and degrade gracefully (show cached data, queue writes for later, return partial results with warnings).

Interactive animation: One lossy downstream service, a 30-second default timeout and 200 threads: watch the waiting climb up three layers.

2. No Global Clock

On a single machine: System.currentTimeMillis() returns a monotonically increasing timestamp. Event A happening before Event B on the same machine has an unambiguous ordering: A's timestamp < B's timestamp.

In a distributed system: each machine has its own clock. Clocks drift (typical drift: 10-100 microseconds per second - machines lose or gain time relative to each other). NTP (Network Time Protocol) synchronizes clocks, but with 1-10ms precision - not microsecond precision. This means: Machine A records Event A at timestamp 10:00:00.001. Machine B records Event B at timestamp 10:00:00.000. Did A happen before B? You can't tell from the timestamps - the clock difference might be drift, not ordering.

Why this is hard: ordering events across machines requires logical clocks (Lamport timestamps, vector clocks) or globally synchronized clocks (Google's TrueTime - hardware-assisted GPS/atomic clock synchronization with bounded uncertainty). Without reliable ordering, determining "which write happened last" (last-writer-wins conflict resolution) is fundamentally ambiguous.

3. No Shared Memory

On a single machine: two threads can read and write the same memory location (with appropriate synchronization - mutexes, atomic operations). Data sharing is instantaneous (memory read: ~100 nanoseconds).

In a distributed system: Machine A and Machine B cannot access each other's memory. Data must be explicitly sent over the network (serialization → network transfer → deserialization). Any "shared state" must be replicated across machines - and keeping replicas consistent is the central challenge of distributed databases (M18.6).

Why this is hard: every piece of shared state becomes a distributed coordination problem. A simple counter (click_count++) on a single machine is an atomic memory operation. A distributed counter across 10 machines requires consensus on the current value, coordination of increments, and handling of network failures during the operation. The operational complexity scales with the amount of shared state.

A Brandon Story: The Fallacy That Cost Four Hours

Interactive animation: Two percent packet loss and a 30-second default timeout: how three layers of services went down.

A team I'd advised built a payment processing service that called three downstream services sequentially: Fraud Detection → Payment Gateway → Notification Service. Each call had a 30-second timeout - the default in their HTTP client library. They'd never changed it because "the network is reliable and latency is low" (Fallacies 1 and 2).

During a cloud provider network event (a transient routing issue affecting ~2% of cross-AZ traffic), calls to the Payment Gateway service experienced 50% packet loss. TCP retransmissions kicked in, but the connection never fully established - the calls hung for the full 30-second timeout before failing.

The cascade: the payment processing service had 200 threads in its thread pool. Each thread handling a payment request waited 30 seconds for the Payment Gateway timeout. Within 60 seconds, all 200 threads were blocked waiting for the Payment Gateway. New incoming requests queued up - no threads available to process them. The service appeared "healthy" (it was accepting TCP connections) but responded to nothing (no threads to serve requests). The load balancer's health check (a simple TCP connect check) passed - so traffic continued flowing to a service that couldn't process anything.

The cascading failure: upstream services calling the payment service also started timing out (their calls to the payment service hung for 30 seconds). THEIR thread pools filled up. Within 5 minutes, three layers of services were down - all because one downstream service's network was experiencing 2% packet loss, and every service had a 30-second default timeout.

The fix (four hours to implement, 30 minutes of downtime):

Aggressive timeouts: Payment Gateway call timeout: 3 seconds (not 30). If the Payment Gateway doesn't respond in 3 seconds, fail fast and retry (or return an error to the user). The user retries in 2 seconds; the total experience is 5 seconds - far better than 30 seconds of hanging.
Circuit breaker (19.5): after 5 consecutive failures to the Payment Gateway, the circuit breaker opens - subsequent calls immediately fail without waiting for a timeout. The circuit breaker retries periodically (every 10 seconds) to check if the Payment Gateway is back. This prevents 200 threads from all waiting 30 seconds simultaneously.
Thread pool bulkhead: separate thread pools for Fraud Detection (50 threads), Payment Gateway (50 threads), and Notification Service (50 threads). If the Payment Gateway pool is exhausted, the Fraud Detection and Notification pools are unaffected - isolation prevents one slow dependency from consuming all resources.
Health check improvement: replace TCP connect health check with a deep health check - an HTTP endpoint that verifies the service can actually process requests (query the database, call a downstream service). If the deep health check fails, the load balancer stops routing traffic to the unhealthy instance.

Lessons:

Default timeouts (30 seconds) are almost always wrong for service-to-service calls. Use 1-5 seconds for synchronous calls. Fail fast, retry, or degrade gracefully.
Thread pool exhaustion from slow dependencies is the #1 cascading failure pattern. Circuit breakers (19.5) and bulkheads prevent it.
TCP connect health checks verify that the OS is accepting connections - not that the application is functioning. Use application-level health checks that exercise the actual request path.
The fallacies are not theoretical. "The network is reliable" and "latency is zero" directly caused a 4-hour cascading outage from a 2% packet loss event affecting one downstream service.
Every distributed system needs: aggressive timeouts, circuit breakers, bulkheaded thread pools, and deep health checks. These are not optimizations - they are survival mechanisms.

Six Takeaways from This Lesson

The Eight Fallacies are assumptions engineers carry from single-machine thinking: reliable network, zero latency, infinite bandwidth, secure network, static topology, one admin, zero transport cost, homogeneous network. All wrong in distributed systems.
Partial failure is the defining characteristic. Machine A is up, Machine B is down - the system is neither fully up nor fully down. Design for graceful degradation, not binary up/down.
No global clock means event ordering across machines is ambiguous. Clock drift (microseconds per second) makes timestamps unreliable for ordering. Logical clocks or TrueTime solve this.
No shared memory means every piece of shared state is a distributed coordination problem. A simple counter becomes a consensus challenge. Minimize shared state.
Default timeouts are dangerous. 30-second defaults cause thread pool exhaustion and cascading failures. Use 1-5 second timeouts for synchronous service calls.
Distributed systems need defensive mechanisms: aggressive timeouts, circuit breakers, bulkheads, retry policies, and deep health checks. These are mandatory, not optional.

Try It

Exercise: Warmup (15 minutes). Check the default timeout for your HTTP client library (Axios, HttpClient, requests, net/http). Is it 30 seconds? Would a 30-second hang in production cause cascading failures in your system? What should it be?

Exercise: Core (25 minutes). Map your system's service call graph (Service A calls B, B calls C and D, etc.). For each call, identify: what happens if that call times out? Does the caller degrade gracefully, retry, or crash? Is there a circuit breaker?

Exercise: Stretch (30 minutes). Read the original "Fallacies of Distributed Computing" paper (link below). For each fallacy, identify one place in your production system where that fallacy could cause an outage. Propose a mitigation.

What's Next

Lesson 19.2 covers network partitions - what happens when machines can't talk to each other, how systems detect partitions, and how they respond.

Back to System Design Fundamentals