Free lesson · System Design Crash Course

Connection Pooling, Keep-Alive, Head-of-Line Blocking

Easy9 min readsystem-designhlddistributed-systems

Lesson 4.7 - Connection Pooling, Keep-Alive, Head-of-Line Blocking

The seventh networking lesson.

The previous five lessons gave you the protocols. This one is about the patterns engineers actually use to apply those protocols efficiently in production. Connection pooling, keep-alive, head-of-line blocking - these aren't separate topics, they're the recurring concerns you'll deal with whenever you're writing or operating networked services.

Here's the practical context: even though TCP and TLS handshakes are expensive (Lessons 4.2 and 4.3), and even though HTTP versions have different multiplexing characteristics (Lessons 4.4-4.6), all of this gets hidden behind HTTP client libraries. You call httpClient.get(url) and you don't see the handshake or the connection management. The library does it for you - well or poorly. This lesson is about understanding what's going on under the hood, so you can configure it well, debug it when it goes wrong, and avoid the common pitfalls.

Let me start with the fundamental pattern.

Why Pool Connections at All?

We established in Lesson 4.2 that opening a TCP+TLS connection is expensive: handshake RTTs, slow start, server-side state. Reusing an existing connection skips all of this.

But you also can't keep every connection open forever. Connections consume server resources. After some inactivity, they should be closed.

The solution: a connection pool. A managed set of open connections that can be reused for multiple requests. When you need to make a request, you grab a connection from the pool; when you're done, you return it. If no connection is available, you create a new one (paying the handshake cost). If you have too many connections, you close some.

flowchart TD
    App["Application code<br/>(makes HTTP requests)"]
    Pool["Connection Pool<br/>(maintains open connections)"]
    C1["Conn 1<br/>(open, idle)"]
    C2["Conn 2<br/>(in use)"]
    C3["Conn 3<br/>(open, idle)"]
    C4["Conn 4<br/>(closed, expired)"]
    Server["Target server"]

    App -->|"request connection"| Pool
    Pool -->|"reuse"| C1
    Pool -.->|"in use"| C2
    Pool -->|"reuse"| C3
    C1 --> Server
    C2 --> Server
    C3 --> Server

    style C1 fill:#34d399,stroke:#10b981,color:#000
    style C2 fill:#fbbf24,stroke:#f59e0b,color:#000
    style C3 fill:#34d399,stroke:#10b981,color:#000
    style C4 fill:#fb7185,stroke:#f43f5e,color:#000

A diagram showing a connection pool. Application code makes HTTP requests. Instead of opening a fresh TCP and TLS connection each time, it asks the connection pool. The pool maintains a set of open connections. Some are idle and ready to reuse  - in green. Some are currently in use by other requests  - in yellow. Some have expired and been closed  - in red. When the application requests a connection, the pool hands out an idle one if available. If all are in use and the pool can grow, a new connection is created with the handshake cost. If the pool is at its limit, the request waits. When done, the connection goes back to the pool as idle. This pattern amortizes the handshake cost across many requests and is essentially mandatory for any service that makes more than a few HTTP calls.

Connection pooling is the pattern for making efficient HTTP clients. Every production HTTP library implements it. The question is whether it's configured correctly.

The Knobs You Need to Know

A typical connection pool has these tunable parameters:

1. Maximum pool size. How many total connections can be in the pool? Usually per-host (e.g., max 20 connections to each unique destination).

2. Maximum idle time. How long can a connection sit idle before being closed? Typical values: 30 seconds to 5 minutes.

3. Maximum lifetime. Total lifetime of a connection, regardless of activity. Often hours.

4. Connection timeout. How long to wait for a new connection to establish before giving up.

5. Request timeout. How long to wait for a request to complete on a connection before giving up.

6. Validation interval. How often to check if idle connections are still alive (TCP keepalive or similar).

Defaults vary by library. Some libraries have surprising defaults - for example, Go's http.DefaultTransport defaults to MaxIdleConns: 100 total across all hosts but only MaxIdleConnsPerHost: 2. If you're calling many distinct hosts heavily, you might hit the per-host limit and accidentally open new connections constantly.

Always check your HTTP client's pool configuration before deploying to production. Defaults often need tuning for high-traffic services.

Keep-Alive in Practice

Keep-alive is the HTTP-level mechanism for keeping connections open. In HTTP/1.1, it's the default (you have to explicitly close); in HTTP/2 and HTTP/3, it's effectively built into the protocol.

A few practical things:

Keep-alive timeouts on both sides. Both the client and the server can decide when to close idle connections. The shorter of the two values wins.

Common server timeouts:

• Nginx: 75 seconds default
• Apache: 5 seconds default
• AWS ALB: 60 seconds default
• HAProxy: 60 seconds default

If your client has a 5-minute keep-alive but the server times out at 60 seconds, the connection will be silently closed by the server after 60 seconds. Your client might try to reuse it and find it's gone - needing to retry.

The "connection closed" race condition. When a server closes a connection, the close packet might arrive after your client has already sent a request on that connection. The client sees the request fail with something like "connection reset" or "broken pipe." This is normal - well-written clients handle this by retrying once on a new connection.

Configure timeouts deliberately. Match your client's keep-alive to the server's keep-alive (or a bit shorter, to be safe). This avoids the close race.

Connection Pool Sizing

A common question: how many connections should I pool?

The answer depends on your traffic and your latency.

Little's Law to the rescue. From Module 2, Lesson 2.2: L = λW. Average number in the system = arrival rate × time spent in the system.

For HTTP requests:

• λ = peak QPS to that backend
• W = average latency per request (including server processing time)
• L = number of simultaneous in-flight requests = connections you need

Example: you make 1000 RPS to a backend, each averaging 50ms. So at any moment, you have ~50 requests in flight. Pool size should be at least 50, probably 75-100 for headroom.

If your pool is smaller than this, requests queue up waiting for available connections - adding latency.

If your pool is larger than necessary, you're wasting memory and possibly running into per-host or system limits.

Practical sizing rule: Use Little's Law to estimate, then add 50-100% headroom for traffic spikes.

Multiplexing Changes the Calculation

For HTTP/2 and HTTP/3, connections are multiplexed. One connection can carry many simultaneous requests. So you need many fewer connections.

In HTTP/1.1: you'd need ~50 connections for 50 simultaneous requests.

In HTTP/2: you might need 1-5 connections total, because each connection can handle many simultaneous streams.

This is a big efficiency win for HTTP/2/3. Fewer connections to manage, less server-side state, simpler operationally.

A subtlety: even with HTTP/2, sometimes you want multiple connections. If one connection's TCP slow start is still ramping up, additional connections fill the gap. Some clients open 2-3 HTTP/2 connections in parallel for this reason. But "6-per-origin" is no longer needed.

Head-of-Line Blocking at Multiple Layers

Let me consolidate what you've learned about head-of-line (HOL) blocking, because it shows up at multiple layers:

1. Application layer HOL (HTTP/1.1).

• Slow request on a connection blocks subsequent requests on that connection.
• Fix: HTTP/2 multiplexing.

2. Transport layer HOL (HTTP/2 over TCP).

• Lost packet blocks delivery of all subsequent packets, including those for different streams.
• Fix: HTTP/3 over QUIC.

3. Operating system HOL (less common but real).

• If your client uses a thread pool and a thread is blocked on a long-running response, other requests waiting for that thread are stalled.
• Fix: async I/O, more threads, separate pools for different request types.

4. Application HOL (your own code).

• If your application processes requests sequentially in some critical section, you've created HOL blocking even with multiplexed transport.
• Fix: parallelism in your application code.

Each layer can be solved independently, but the fixes have to compose. HTTP/3 fixes transport HOL but doesn't help if your application code has its own HOL.

Common Mistakes

A few patterns that beginners (and some seniors) get wrong:

Mistake 1: Not Pooling at All

The classic mistake: HTTP client without connection pooling, opening a fresh connection per request. Symptoms: high latency for each request (paying handshake), TIME_WAIT exhaustion, "too many open files" errors, or general bad performance.

Fix: configure connection pooling. Most modern HTTP libraries do this by default, but it's worth verifying.

Mistake 2: Pool Too Small

Pool sized for normal load, not peak. Symptoms: requests queue up during traffic spikes, latency suddenly spikes.

Fix: size for peak with headroom. Monitor pool utilization.

Mistake 3: Pool Too Large

Trying to pool 1000 connections to a backend that can only handle 100 well. Symptoms: backend gets overwhelmed; cascade failures.

Fix: respect backend capacity. Use circuit breakers and bulkheads (Module 1, Lesson 1.5) to limit concurrency.

Mistake 4: Connection Reuse Across Requests Within a Single Logical Operation

Subtle but real. If you make 3 sequential HTTP calls to do one logical operation, and each opens its own connection because of poor library defaults, you've made the operation 3x slower than necessary.

Fix: ensure connection reuse across related calls in the same operation.

Mistake 5: Different Behavior Across Environments

Common: production uses connection pooling correctly, but tests / staging use a different client config without pooling. Then you discover the pool doesn't work in production when you actually deploy.

Fix: same client config across all environments.

Mistake 6: Forgetting Keep-Alive in CDN/Proxy Layer

The path is client → load balancer → backend. If keep-alive works between client and LB but not between LB and backend, you're still paying handshakes for every backend call.

Fix: configure keep-alive at every hop.

Practical Configuration Examples

Here are sensible defaults for various contexts (just rough starting points).

Go HTTP client for service-to-service:

Transport: &http.Transport{
  MaxIdleConns:        100,
  MaxIdleConnsPerHost: 20,
  IdleConnTimeout:     90 * time.Second,
  TLSHandshakeTimeout: 10 * time.Second,
}

Node.js https.Agent:

new https.Agent({
  keepAlive: true,
  keepAliveMsecs: 60000,
  maxSockets: 50,
  maxFreeSockets: 10,
  timeout: 60000,
})

Java OkHttp:

new ConnectionPool(20, 5, TimeUnit.MINUTES);

Python httpx (recommended over requests for performance):

httpx.Client(
  limits=httpx.Limits(max_connections=100, max_keepalive_connections=20),
  timeout=30,
)

These aren't universally correct - tune for your traffic. But starting with sensible defaults beats starting with the library defaults, which are often very conservative.

How to Tell If You Have a Connection Problem

Symptoms of poor connection management:

1. High variance in latency. Most requests are fast; occasional ones are slow. The slow ones are paying handshake costs.

2. Latency drops dramatically with concurrency. First request is slow; later requests (reusing connection) are fast.

3. "Too many open files" or "TIME_WAIT" errors. Server-side socket exhaustion.

4. Connection refused errors during peak. Backend can't accept more connections.

5. High CPU on cryptographic operations. TLS handshakes use significant CPU; if you're handshaking constantly, this dominates.

Diagnostics tools:

• ss -s (Linux) - socket statistics, including TIME_WAIT counts
• netstat -an | wc -l - total open sockets
• lsof -i - open network connections
• HTTP client metrics - most libraries expose pool stats; export them

When numbers look wrong (too many connections, too many in TIME_WAIT, high handshake rate), it's usually a config issue.

A Real Story: Brandon at S3

When I was working at S3, we had an internal service that did health checks against thousands of backend nodes. The service made HTTPS calls - small, frequent, parallel.

For months it ran fine. Then we expanded to a new region, and suddenly the service started failing. Investigation showed it was hitting connection limits on the load balancer in front of the backend nodes.

The issue: the HTTP client we were using had connection pooling, but the per-host pool was small (the library default). For most operations, this was fine - but the new region's load balancer happened to be a single host name that all health checks went to. So we needed many connections to that one host, but the per-host limit blocked us.

Fix: bumped per-host pool size. Service started behaving correctly.

The takeaway: library defaults are conservative because they're trying to be safe for many use cases. For high-traffic production services, you almost always need to tune them. Treat HTTP client configuration as part of your design.

Six Takeaways from This Lesson

1. Connection pooling is essential for any service that makes HTTP calls. Without it, you pay handshake cost on every request.

2. Size your pool using Little's Law: peak QPS × average latency = simultaneous requests = pool size.

3. HTTP/2 and HTTP/3 multiplexing dramatically reduce the connections you need. Per-host limits become much less of a concern.

4. Keep-alive timeouts on client and server need to match. Mismatches cause "connection reset" errors.

5. Head-of-line blocking exists at multiple layers - HTTP, transport, OS, application. Address each independently.

6. Library defaults are conservative. Always verify and tune connection pool settings for production.

Try It

Exercise: Warmup (10 minutes). Check the HTTP client defaults for whatever library you use most (Python requests, Go http, Node.js, Java OkHttp). What's the default pool size? Idle timeout?

Exercise: Core (20 minutes). Use ss -s or equivalent on a production-like system. Note the connection counts in various states. Are there a lot of TIME_WAIT sockets? Could indicate excessive connection churn.

Exercise: Stretch (40 minutes). Read about a specific production incident caused by connection issues. Search for "connection pool exhaustion" or "TIME_WAIT exhaustion" postmortems. Note what went wrong and the fix.

What's Next

Lesson 4.8 is the synthesis - we'll decompose a typical 200ms request and see where the time actually goes. With all the protocol knowledge from this module, the latency breakdown should make complete sense.

Back to System Design Crash Course