Free lesson · System Design Fundamentals

What Are Load Balancers?

Easy11 min readsystem-designhlddistributed-systems

Lesson 4.1 - What Are Load Balancers?

The first lesson of Module 4.

Imagine you've built an API. It runs on a single server. The server handles 500 requests per second comfortably. Your application grows. Traffic reaches 2,000 requests per second. The server's CPU is pegged at 100%, response times spike from 50ms to 4 seconds, and users start seeing timeouts. The obvious fix: add more servers. But now you have a new problem - how do clients know which server to talk to? You can't give each user a different URL. You can't ask them to manually pick a server. You need something in the middle that accepts all traffic on one address and distributes it across your servers. That something is a load balancer.

The Core Concept

A load balancer is a component that sits between clients and a group of backend servers (often called a server pool, backend pool, or upstream group). It accepts incoming requests on a single public address and forwards each request to one of the backend servers based on an algorithm (covered in 4.2). To the client, it looks like one server. Behind the load balancer, there might be 3 servers or 3,000.

flowchart TD
    C1["Client A<br/>(browser)"]
    C2["Client B<br/>(mobile app)"]
    C3["Client C<br/>(API consumer)"]
    
    LB["Load Balancer<br/>Public IP: 54.23.100.1<br/>Accepts all traffic"]
    
    S1["Server 1<br/>Private IP: 10.0.1.10<br/>Handling 500 req/s"]
    S2["Server 2<br/>Private IP: 10.0.1.11<br/>Handling 480 req/s"]
    S3["Server 3<br/>Private IP: 10.0.1.12<br/>Handling 520 req/s"]
    
    C1 --> LB
    C2 --> LB
    C3 --> LB
    
    LB --> S1
    LB --> S2
    LB --> S3

    style LB fill:#fbbf24,stroke:#f59e0b,color:#000
    style S1 fill:#34d399,stroke:#10b981,color:#000
    style S2 fill:#34d399,stroke:#10b981,color:#000
    style S3 fill:#34d399,stroke:#10b981,color:#000

A diagram showing a load balancer distributing traffic across three backend servers. Three clients - a browser, a mobile app, and an API consumer - all send requests to the load balancer's single public IP address: 54.23.100.1. The clients don't know or care that multiple servers exist behind it. The load balancer receives each request and forwards it to one of three backend servers. Each server has a private IP address (10.0.1.10, 10.0.1.11, 10.0.1.12) that's not directly accessible from the internet. The servers are handling roughly equal traffic - 500, 480, and 520 requests per second respectively. Together they handle approximately 1,500 requests per second - three times what any single server could handle. If Server 2 crashes, the load balancer detects this via health checks and stops sending traffic to it. Clients 1 and 3 continue being served by Servers 1 and 3 with no interruption. The user never sees the failure. When a replacement Server 2 comes online, the load balancer detects it and resumes sending traffic to it.

What the load balancer gives you:

1. Horizontal scalability. Need to handle more traffic? Add more servers behind the load balancer. No client-side changes. No DNS updates. No application code changes. The load balancer distributes traffic across whatever servers are in the pool. This is horizontal scaling (M2.1) made practical.

2. High availability. If a server dies, the load balancer detects it (via health checks) and stops routing traffic to it. Clients are served by the remaining healthy servers. No manual intervention. No user-visible downtime. This directly addresses the SPOF problem from M2.4 - a single server is a SPOF; multiple servers behind a load balancer are not.

3. Transparent maintenance. Need to deploy a new version? Pull one server out of the pool, deploy to it, add it back. Repeat for each server. Clients never see the deployment because the load balancer routes around the server that's being updated. This is the foundation of rolling deployments (M40).

4. Flexible resource usage. Different servers can have different capacities. A powerful server can receive more traffic (weighted algorithms - 4.2). New servers can be added during traffic spikes and removed when traffic subsides (auto-scaling).

Where Load Balancers Sit

Load balancers don't appear in just one place. A production architecture often has them at multiple layers:

flowchart TD
    Internet["Internet<br/>(users)"]
    
    GLB["Global Load Balancer<br/>(DNS / CDN / Anycast)<br/>Routes to nearest region"]
    
    subgraph Region1["Region: us-east-1"]
        ELB1["External LB<br/>(ALB / Nginx)<br/>Routes to API servers"]
        API1["API Servers<br/>(3 instances)"]
        ILB1["Internal LB<br/>Routes to services"]
        SVC1["Microservices<br/>(auth, users, orders)"]
        DBLB1["DB Proxy<br/>(PgBouncer / RDS Proxy)<br/>Connection pooling"]
        DB1["Database<br/>(primary + replicas)"]
    end
    
    Internet --> GLB --> ELB1 --> API1 --> ILB1 --> SVC1 --> DBLB1 --> DB1

    style GLB fill:#a78bfa,stroke:#8b5cf6,color:#000
    style ELB1 fill:#fbbf24,stroke:#f59e0b,color:#000
    style ILB1 fill:#fbbf24,stroke:#f59e0b,color:#000
    style DBLB1 fill:#fbbf24,stroke:#f59e0b,color:#000

A multi-layer load balancing diagram. At the top, internet traffic hits a global load balancer - this could be DNS-based routing, a CDN like Cloudflare, or anycast (covered in lessons 4.3 and 4.4). It routes the user to the nearest region. Within a region (us-east-1 in this example), traffic hits an external load balancer - an AWS ALB or Nginx instance - which distributes requests across multiple API server instances. The API servers then talk to downstream microservices through an internal load balancer. The microservices connect to the database through a database proxy like PgBouncer or RDS Proxy, which acts as a connection-pooling load balancer - multiplexing many short-lived application connections into a smaller number of long-lived database connections. That's four layers of load balancing in a single architecture: global routing, external API balancing, internal service balancing, and database connection pooling. Each layer solves a different distribution problem.

Layer by layer:

Global load balancer (4.3, 4.4): distributes traffic across regions or data centers. Uses DNS (Route 53 geographic routing), anycast (Cloudflare), or a dedicated global load balancer (Google Cloud Global LB). The decision is usually "which region is closest to this user?"

External load balancer: distributes internet-facing traffic across your API servers. This is the most visible load balancer - the one clients connect to. AWS ALB, Google Cloud Load Balancer, or self-managed Nginx/HAProxy. Handles TLS termination (decrypts HTTPS so backend servers deal with plain HTTP), routes by URL path, and checks backend health.

Internal load balancer: distributes service-to-service traffic within your infrastructure. API servers calling authentication service, users service, orders service - each behind its own internal load balancer. Often implemented as a service mesh (Envoy sidecar) or a Kubernetes Service (which is an internal LB built into the platform - M39).

Database proxy / connection pooler: distributes database connections across read replicas and pools connections to the primary. PgBouncer (PostgreSQL), RDS Proxy (AWS managed), or ProxySQL (MySQL). This solves the "too many connections" problem - 100 application instances each opening 10 connections = 1,000 connections to the database. The proxy multiplexes them into 50 persistent connections. This connects directly to M27.3's Lambda concurrency discussion - Lambda scales to thousands of instances, each needing a database connection.

Layer 4 vs Layer 7 Load Balancers

This distinction appeared in M3.1 (OSI model) and is critical for load balancer selection:

Layer 4 (Transport layer) load balancers operate on TCP/UDP - they see source IP, destination IP, source port, and destination port. They forward packets without inspecting the content. They're fast (minimal processing per packet) but blind to the application protocol - they can't route by URL path, inspect HTTP headers, or make decisions based on cookies.

Layer 7 (Application layer) load balancers operate on HTTP/HTTPS - they understand the request content. They can route by URL path (/api/v1/* to fleet A, /api/v2/* to fleet B), by HTTP header (route mobile clients differently from desktop), by cookie (sticky sessions), and by request body content. They can also terminate TLS, compress responses, cache static content, and rewrite URLs. They're slower (must parse the full HTTP request) but much more capable.

The default choice for most web applications: Layer 7. The ability to route by path, terminate TLS, and inspect HTTP headers is essential for modern APIs and microservices. Choose Layer 4 only when you need raw TCP/UDP forwarding (gaming servers, database proxies, extremely high throughput where the ~5ms L7 overhead matters).

Health Checks: How the Load Balancer Knows What's Alive

A load balancer that sends traffic to dead servers is worse than useless - it's actively causing failures. Health checks are how the load balancer knows which backends are healthy:

Active health checks: the load balancer periodically sends a request to each backend (e.g., GET /health every 10 seconds). If a backend doesn't respond (or responds with an error) for N consecutive checks, the load balancer marks it unhealthy and stops routing traffic to it. When it starts responding again, the LB marks it healthy and resumes routing.

Passive health checks: the load balancer monitors real traffic. If a backend returns 5 consecutive 5xx errors, the LB marks it unhealthy. No separate health check requests - the real traffic IS the check.

What a good health check endpoint does: not just "return 200." A meaningful health check verifies the backend can actually serve requests - can it reach the database? Is the disk not full? Is the application thread pool not exhausted? A server that returns 200 on /health but can't query the database is lying about its health.

sequenceDiagram
    participant LB as Load Balancer
    participant S1 as Server 1 (healthy)
    participant S2 as Server 2 (failing)
    
    loop Every 10 seconds
        LB->>S1: GET /health
        S1-->>LB: 200 OK
        LB->>S2: GET /health
        S2-->>LB: 503 (DB connection failed)
    end
    
    Note over LB: S2 failed 3 consecutive checks.<br/>Marked unhealthy. Traffic routed<br/>only to S1 until S2 recovers.
    
    LB->>S1: Forward client request
    S1-->>LB: 200 OK (response)
    
    Note over S2: S2 recovers. Passes 2 consecutive<br/>health checks. Marked healthy again.
    
    LB->>S2: Resume forwarding traffic

A sequence diagram showing health checks in action. The load balancer sends a GET request to the /health endpoint of each server every 10 seconds. Server 1 responds with 200 OK - it's healthy. Server 2 responds with 503 - its database connection has failed, so it reports itself as unhealthy. After 3 consecutive failed health checks (30 seconds), the load balancer marks Server 2 as unhealthy and stops sending traffic to it. All client requests now go to Server 1 only. Meanwhile, Server 2 continues receiving health check pings. When it recovers (database connection restored), it starts responding with 200 OK. After 2 consecutive successful checks, the load balancer marks it healthy again and resumes sending traffic to it. The entire recovery is automatic - no human intervention, no restart, no deployment. The health check is the mechanism that makes load balancers self-healing.

Health check configuration parameters:

The unhealthy/healthy thresholds prevent flapping. A single failed check doesn't remove a server (could be a network blip). Three consecutive failures signal a real problem. Similarly, a single successful check doesn't re-add a recovering server - it must pass multiple checks to confirm it's truly back.

A Brandon Story: The Health Check That Lied

A team I'd advised had a health check endpoint that returned 200 OK if the application process was running - nothing more. The server's disk filled up (application logs consumed all space). The application couldn't write to disk, so every request that needed to write a log or temp file failed with an I/O error. But /health returned 200 because the process was technically alive and the health check didn't write anything.

The load balancer kept routing traffic to this server for 45 minutes. Every request routed to it failed. With 3 servers in the pool, ~33% of all requests were failing - an effective partial outage caused by a health check that didn't check the right things.

The fix: the health check endpoint was rewritten to verify three things:

1. The application process is running (trivial - if it responds, it's running)
2. The database is reachable (execute a lightweight query: SELECT 1)
3. Disk space is available (check free space > 100 MB)

If any check failed, the endpoint returned 503. The load balancer would remove the server within 30 seconds, routing traffic to the healthy servers while the team investigated.

After the fix, they also added a /ready endpoint (separate from /health) for the Kubernetes readiness probe - indicating "I can accept traffic" vs "I am alive." A server can be alive (healthy) but not ready (still warming up, loading cache, running migrations). Kubernetes uses both: liveness probe (/health) to decide whether to restart the pod, readiness probe (/ready) to decide whether to send it traffic.

Lessons:

• A health check that doesn't verify the dependencies is worse than no health check - it provides false confidence
• Check what matters: database connectivity, disk space, external service reachability - not just "is the process alive?"
• Separate health (alive) from readiness (can serve traffic) - they're different questions
• 33% of requests failing for 45 minutes is a severity-1 incident caused by a 5-line health check endpoint

Six Takeaways from This Lesson

1. A load balancer distributes traffic across multiple servers behind a single address. Clients see one endpoint; the LB handles distribution.

2. Load balancers provide scalability (add servers for more capacity), availability (route around failures), and transparent maintenance (rolling deploys without downtime).

3. They appear at every layer: global (DNS/anycast), external (internet-facing API), internal (service-to-service), and database (connection pooling).

4. Layer 4 (TCP/UDP) is fast but blind to content. Layer 7 (HTTP) can route by path, headers, and cookies. Layer 7 is the default for web applications.

5. Health checks are non-negotiable. Active checks (periodic pings) and passive checks (monitoring real traffic) both have a place. Check the dependencies, not just the process.

6. Separate health from readiness. "Alive" and "can serve traffic" are different states - both need their own check.

Try It

Exercise: Warmup (15 minutes). Draw an architecture diagram for a web application with 3 API servers behind a load balancer. Label the public IP (on the LB) and private IPs (on the servers). Add a health check path. Identify: what happens when Server 2 dies?

Exercise: Core (25 minutes). Your application has two types of traffic: /api/* (JSON APIs, latency-sensitive) and /uploads/* (large file uploads, throughput-sensitive). Design the load balancer configuration: do you use Layer 4 or Layer 7? Do you use one LB or two? How do you route each traffic type?

Exercise: Stretch (45 minutes). Set up Nginx as a reverse proxy load balancer locally. Configure it with 2 backend servers (two instances of a simple HTTP server on different ports). Configure a health check. Kill one backend and observe Nginx routing all traffic to the surviving backend. Then restart the dead backend and observe traffic resuming.

What's Next

Lesson 4.2 covers the algorithms - how the load balancer decides which server gets each request. Round robin, weighted round robin, least connections, IP hash, random, and EWMA.

Back to System Design Fundamentals