AlgoInsight

System Design

How Load Balancers Actually Work

Every large system has one sitting quietly out front, deciding where each request goes. Here's what a load balancer really does - the algorithms, the health checks, the sticky sessions, and how it avoids becoming the very thing it prevents.

By Admin2026-06-1814 min readload-balancersystem-designscalabilityhigh-availabilitynetworkingdistributed-systems

The moment a system grows beyond what one server can handle, you run multiple servers - and immediately face a new question: when a request comes in, which server should handle it? Answer that badly and you get some servers overwhelmed while others sit idle, slow responses, and outages when a server dies and requests keep flowing to it anyway.

The load balancer is the answer. It sits in front of your pool of servers and distributes incoming requests across them, so the work is shared evenly and no single server becomes a bottleneck. It's one of the most fundamental building blocks in system design - nearly every large-scale system has one. This guide explains how load balancers actually work: how they distribute traffic, the algorithms they use, how they detect failed servers, how they handle sessions, and how they avoid becoming a single point of failure themselves.

What a load balancer does

At its core, a load balancer is a traffic director. Clients send their requests to the load balancer, and the load balancer forwards each one to one of the backend servers in its pool. To the client, it looks like they're talking to a single server; in reality, their requests are being spread across many.

graph LR
    C1[Client] --> LB
    C2[Client] --> LB
    C3[Client] --> LB
    LB[Load Balancer] --> S1[Server 1]
    LB --> S2[Server 2]
    LB --> S3[Server 3]
    LB --> S4[Server 4]

This diagram shows a load balancer in action. Multiple clients send their requests to a single load balancer, which then distributes those requests across a pool of four backend servers. Each client sees one address, but their requests are spread across all four servers behind it.

This delivers three essential benefits:

Layer 4 vs Layer 7 load balancing

Load balancers operate at different levels of the networking stack, and the difference shapes what they can do.

A Layer 4 (transport-level) load balancer makes its decisions based on network information - IP addresses and ports - without inspecting the actual content of the request. It simply sees a connection coming in and forwards it to a server. Because it doesn't look inside the request, it's extremely fast and efficient, but it can't make content-based routing decisions.

A Layer 7 (application-level) load balancer looks at the actual content of requests - the HTTP method, URL path, headers, and cookies. This lets it make smarter decisions: routing /images/* requests to servers optimized for images and /api/* requests to API servers, for example, or routing based on which user is making the request. The cost is a bit more processing overhead, since it has to parse each request.

Layer 4 Layer 7
Routes based on IP address, port URL, headers, cookies, content
Speed Very fast (no content inspection) Slightly slower (parses requests)
Flexibility Limited (no content awareness) High (content-based routing)
Best for Raw speed, simple distribution Smart routing, mixed workloads

Most modern web systems use Layer 7 load balancers for their flexibility, but Layer 4 remains valuable where raw throughput matters more than routing intelligence.

The algorithms: how it decides

The heart of a load balancer is the algorithm it uses to choose which server gets each request. Several are common, each with its own strengths.

Round robin is the simplest: cycle through the servers in order - server 1, server 2, server 3, server 4, then back to server 1. It's easy to understand and works well when all servers are roughly equal and all requests take roughly the same amount of work. Its weakness is that it ignores how busy each server actually is.

Least connections sends each new request to the server currently handling the fewest active connections. This adapts to reality: if one server is bogged down with long-running requests, it has more open connections, so new requests go elsewhere. It's a better choice when requests vary widely in how long they take.

Weighted round robin assigns each server a weight based on its capacity, sending more requests to more powerful servers. If server 1 is twice as powerful as server 2, it gets twice the traffic. This handles fleets where the servers aren't identical.

IP hash routes a given client (based on a hash of their IP address) consistently to the same server. This is useful when you want a particular client to keep hitting the same server - which connects to the sticky-session topic below.

Least response time combines connection count with how fast each server is currently responding, sending requests to the server that's both least busy and quickest. It's more sophisticated and adapts well, at the cost of needing more information about each server.

Here's the logic of the two most common algorithms, simplified:

class LoadBalancer:
    def __init__(self, servers):
        self.servers = servers
        self.index = 0
        self.connections = {s: 0 for s in servers}

    def round_robin(self):
        server = self.servers[self.index]
        self.index = (self.index + 1) % len(self.servers)
        return server

    def least_connections(self):
        # pick the server with the fewest active connections
        return min(self.connections, key=self.connections.get)

lb = LoadBalancer(["A", "B", "C"])
print("Round robin:", [lb.round_robin() for _ in range(6)])

# simulate B being busy with 3 open connections
lb.connections["B"] = 3
print("Least connections picks:", lb.least_connections())

Round robin cycles A, B, C, A, B, C regardless of load; least connections looks at the actual state and avoids the busy server. The right algorithm depends on whether your requests are uniform (round robin is fine) or variable (least connections or least response time adapt better).

Health checks: routing around failure

A load balancer's high-availability benefit depends entirely on knowing which servers are actually working. It learns this through health checks - periodic probes sent to each server to confirm it's alive and responding correctly.

A health check might be as simple as opening a connection, or as thorough as calling a dedicated /health endpoint that returns 200 OK only when the server and its dependencies are functioning. The load balancer sends these checks on a schedule. When a server starts failing them, the load balancer marks it unhealthy and stops sending it traffic - automatically routing around the failure. When the server starts passing checks again, it's returned to the pool.

This is the mechanism behind "if a server dies, traffic goes to the healthy ones." Without health checks, a load balancer would keep cheerfully sending requests to a dead server, and users would hit errors. Tuning the health check - how often to probe, how many failures before marking a server dead - involves the same trade-off as any failure detector: too aggressive and a briefly-slow server gets pulled unnecessarily; too lenient and a dead server keeps receiving traffic.

Tip: A good health check verifies that the server can actually do its job, not just that it's powered on. A server might respond to a basic ping while its database connection is broken - so it would fail real requests. A health check that exercises the real dependencies catches this; a shallow one doesn't.

Sticky sessions: when a client needs the same server

By default, a load balancer might send a given client to a different server on each request. Usually that's fine - but sometimes a client needs to keep hitting the same server because that server holds session-specific state (an in-progress shopping cart in memory, say). Sticky sessions (also called session affinity) make the load balancer consistently route a particular client to the same backend server, often by using a cookie or the client's IP.

Sticky sessions solve the immediate problem but carry a real cost: they undermine even distribution (one server can accumulate many sticky clients), and they reintroduce a failure problem (if that server dies, its clients lose their session state). For this reason, the better long-term solution is usually to make servers stateless - store session state in a shared external store (like a cache or database) rather than in any one server's memory - so any server can handle any request and sticky sessions become unnecessary.

Warning: Sticky sessions are often a sign that state is living in the wrong place. They work, but they fight against the scalability and resilience a load balancer is supposed to provide. When you can, push session state into a shared store and keep your servers stateless - then any server can serve any request and the load balancer is free to distribute purely for efficiency.

The load balancer itself: avoiding a single point of failure

Here's a question that catches people out: if all traffic flows through the load balancer, isn't the load balancer itself a single point of failure? If it goes down, the whole system goes down - no matter how many healthy servers are behind it.

This is a real problem, and it's solved by making the load balancer redundant. Production systems run multiple load balancers in a failover configuration: if the active load balancer fails, a standby immediately takes over. Often this is paired with DNS-level techniques or a floating IP address that can move to the backup. The principle is the same one the load balancer applies to your servers, turned back on itself - no single component, including the traffic director, should be able to take down the whole system.

At the largest scale, this extends to global server load balancing, where load balancers in multiple geographic regions direct users to the nearest or healthiest data center - distributing traffic not just across servers but across the whole world, and surviving the loss of an entire region.

Interview: When you put a load balancer in a system design, a strong follow-up to anticipate is "isn't that a single point of failure?" The answer that signals depth: "Yes, which is why I'd run the load balancer itself redundantly - an active-standby pair with automatic failover - so the component preventing single points of failure isn't one itself." Naming this before being asked is exactly the kind of proactive failure-thinking interviewers reward.

DNS load balancing: distribution before the load balancer

There's a layer of load balancing that happens even before traffic reaches a load balancer: DNS load balancing. When a client looks up your domain name, the DNS system can return different server addresses to different clients - rotating through them, or returning the address of the nearest or healthiest data center. This spreads traffic at the very first step, the name lookup, and is how traffic gets distributed across multiple regions or multiple load balancers in the first place.

DNS load balancing is coarse - DNS responses are cached by clients and resolvers, so changes propagate slowly, and it can't react instantly to a server failing. That's why it's typically used alongside dedicated load balancers rather than instead of them: DNS spreads traffic across regions and data centers at a high level, and a load balancer within each region does the fast, fine-grained, health-aware distribution across servers.

Connection draining: removing servers gracefully

A practical detail that matters in real operations: what happens when you need to remove a server from the pool - to deploy a new version, or to retire it? If you yank it out immediately, any requests it's currently handling get dropped, and users see errors. Connection draining (also called graceful shutdown) solves this: the load balancer stops sending the server new requests but lets it finish the ones it's already handling before fully removing it. This is what makes zero-downtime deployments possible - you drain a server, update it, return it to the pool, and move to the next, with users never noticing.

This connects to a broader point: load balancers are central to how modern systems deploy without downtime. In a rolling deployment, you update servers a few at a time - draining each, updating it, health-checking it, and returning it to the pool before moving on. The load balancer's health checks ensure traffic only flows to servers that are actually ready, and connection draining ensures in-flight requests aren't dropped. The same machinery that distributes everyday traffic is what lets you ship new code to a live system safely.

Global load balancing in more depth

For systems serving users worldwide, a single pool of servers in one location isn't enough - a user in Asia shouldn't have every request cross an ocean to a data center in North America. Global server load balancing (GSLB) distributes traffic across data centers in multiple regions, directing each user to the closest or healthiest one, usually via DNS-level decisions. This cuts latency by keeping users geographically near the servers handling their requests, and it provides disaster resilience: if an entire region goes offline, GSLB routes its traffic to the others, and the system survives the loss of a whole data center. Local load balancers then handle distribution within each region. The result is a hierarchy - global balancing across regions, local balancing across servers - that keeps a worldwide system both fast and resilient.

Hardware vs software load balancers

Load balancers come in two broad forms. Hardware load balancers are dedicated physical appliances - powerful and reliable, but expensive and less flexible. Software load balancers run as programs on standard servers (or as managed cloud services) and are far more common today because they're flexible, cost-effective, and scale elastically with demand. Cloud providers offer load balancing as a managed service, handling the redundancy and scaling for you. For most systems, a software or managed load balancer is the practical choice.

The bottom line

A load balancer is the traffic director that makes horizontal scaling and high availability possible. It distributes requests across a pool of servers using an algorithm suited to your workload, uses health checks to route around failed servers automatically, handles session affinity when needed (though stateless servers are better), and must itself be made redundant so it doesn't become the single point of failure it exists to prevent.

It's a deceptively simple-looking component doing load-bearing work, and it connects directly to the reverse proxy and API gateway roles that often sit alongside or overlap with it. We cover load balancing, and how all these front-door components fit together, across System Design Fundamentals.

Go deeper

More articles