Free lesson · Computer Networking

Summary statistics (total connections by state)

Easy11 min readnetworkingcomputer-networkstcp-ip

The Operational Side of TCP

The previous topics in this module covered how TCP works in theory - handshakes, reliability, flow control, congestion control, connection termination, and limitations. This topic covers how TCP works in practice: the kernel parameters you tune, the diagnostic tools you use, the problems you encounter, and the decisions you make when TCP is not the right choice.

Every production system runs on TCP. Web servers, databases, message queues, caches, load balancers, API gateways - all use TCP as their transport. Understanding how to tune and troubleshoot TCP is not optional for engineers who operate services at scale. A misconfigured TCP parameter can cause mysterious latency spikes, connection failures, or throughput limitations that are invisible to application-level monitoring.

Essential TCP Parameters on Linux

Linux exposes dozens of TCP tuning parameters through the sysctl interface. These are the ones that matter most in production:

Connection Parameters

net.ipv4.tcp_max_syn_backlog (default: 1024 on many systems) The maximum number of half-open connections (SYN-RECEIVED state) the kernel will queue. Under SYN flood attacks or burst traffic, this queue fills up and new SYN packets are dropped. Increase to 4096 or 8192 for busy servers.

net.core.somaxconn (default: 4096) The maximum size of the accept queue - completed connections waiting for the application to call accept(). If the application is slow to accept connections, this queue fills and new connections are dropped or rejected. Many web servers (Nginx, Node.js) set their own listen backlog; the effective value is min(application_backlog, somaxconn).

net.ipv4.tcp_keepalive_time (default: 7200 seconds = 2 hours) How long a connection must be idle before TCP sends keepalive probes. The 2-hour default is far too long for most applications - load balancers, firewalls, and NAT devices typically time out idle connections after 5-15 minutes. Set to 60-300 seconds for production servers behind load balancers to detect dead connections before the LB does.

net.ipv4.tcp_keepalive_intvl (default: 75 seconds) Interval between keepalive probes after the first probe is sent.

net.ipv4.tcp_keepalive_probes (default: 9) Number of keepalive probes before the connection is declared dead.

Buffer and Window Parameters

net.ipv4.tcp_rmem and net.ipv4.tcp_wmem Three-value settings (min, default, max) for TCP receive and send buffer sizes in bytes. The default is typically 4096 131072 6291456 (4 KB min, 128 KB default, 6 MB max). For high-BDP connections (cross-region replication, large file transfers), increase the maximum to accommodate the bandwidth-delay product. TCP autotuning adjusts the actual buffer size between the default and maximum based on connection conditions.

net.core.rmem_max and net.core.wmem_max Hard ceiling for socket buffer sizes. Individual sockets cannot exceed these values even if tcp_rmem/tcp_wmem specifies a higher maximum. Set these at least as high as the tcp_rmem/tcp_wmem maximum values.

Connection Reuse and TIME_WAIT

net.ipv4.tcp_tw_reuse (default: 0 or 2) Enables reuse of TIME_WAIT sockets for new outgoing connections when the timestamp is newer. Safe to enable on servers that make many outbound connections (API gateways, proxies). Value 2 (enabled for loopback and outgoing) is the default in newer kernels.

net.ipv4.ip_local_port_range (default: "32768 60999") The range of ephemeral ports available for outgoing connections. Widen to "1024 65535" on servers with high outbound connection rates to delay port exhaustion.

Diagnosing TCP Issues with ss and netstat

The ss command (socket statistics) is the modern replacement for netstat on Linux. It provides detailed information about TCP connections, states, and parameters.

# Summary statistics (total connections by state)
ss -s

# All TCP connections with process info
ss -tnp

# Connections in specific states
ss -tn state established
ss -tn state time-wait
ss -tn state close-wait

# Detailed connection info (timer, window, congestion)
ss -tin

# Count TIME_WAIT sockets
ss -tn state time-wait | wc -l

What Each TCP State Tells You

Practical Debugging Workflow

When diagnosing a TCP issue, start with the macro view and drill down:

Step 1: ss -s - Get overall connection counts by state. Identify anomalies (unusually high TIME_WAIT, growing CLOSE_WAIT, many SYN_RECV).

Step 2: ss -tnp state <problematic_state> - List specific connections in the anomalous state with their process IDs. Identify which application is responsible.

Step 3: ss -tin dst <destination> - Examine detailed parameters for connections to a specific destination. Check RTT (rtt:), congestion window (cwnd:), retransmissions (retrans:), and send buffer usage.

Step 4: tcpdump or Wireshark - Capture actual packets for deep analysis. Filter by the specific connection to see handshake timing, retransmissions, window size changes, and RST segments.

graph TD
    subgraph Debug["TCP Debugging Workflow"]
        direction TB
        S1["ss -s<br/>Connection state<br/>summary"] --> S2["ss -tnp state X<br/>List connections<br/>in problem state"]
        S2 --> S3["ss -tin dst Y<br/>Detailed params<br/>for specific dest"]
        S3 --> S4["tcpdump / Wireshark<br/>Packet-level<br/>analysis"]
    end

    style S1 fill:#4f46e5,stroke:#4338ca,color:#fff
    style S2 fill:#7c3aed,stroke:#6d28d9,color:#fff
    style S3 fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style S4 fill:#a78bfa,stroke:#8b5cf6,color:#fff

This diagram shows the TCP debugging workflow. Start with ss -s for a connection state summary - identify anomalies like high TIME_WAIT or growing CLOSE_WAIT. Then use ss -tnp with a state filter to list specific connections in the problematic state with their process IDs. Next, use ss -tin with a destination filter to examine detailed TCP parameters like RTT, congestion window, and retransmissions for specific connections. Finally, use tcpdump or Wireshark for packet-level analysis to see handshake timing, retransmissions, and window behavior.

SYN Floods and Mitigation

A SYN flood is a denial-of-service attack where an attacker sends a high volume of SYN segments from spoofed IP addresses. The server allocates resources for each SYN-RECEIVED connection and sends SYN-ACKs to the spoofed addresses, which never respond with the final ACK. The server's SYN queue fills, and legitimate connections are rejected.

SYN cookies are the primary kernel-level defense. When enabled (net.ipv4.tcp_syncookies = 1), the server does not allocate any state for incoming SYN segments. Instead, it encodes the connection information (client IP, port, MSS, timestamp) into the SYN-ACK's sequence number using a cryptographic hash. If the client completes the handshake with a valid ACK, the server can reconstruct the connection state from the ACK's acknowledgment number. This eliminates the SYN queue entirely under attack conditions.

SYN cookies have a downside: TCP options from the original SYN (window scaling, SACK, timestamps) cannot be stored, so connections established via SYN cookies may have reduced performance. Modern Linux implementations activate SYN cookies only when the SYN backlog overflows, preserving full TCP options for normal traffic while protecting against floods.

Cloud environments add additional layers of SYN flood protection: AWS Shield, Cloudflare's DDoS mitigation, and GCP Cloud Armor filter attack traffic at the network edge before it reaches your servers.

Common Production TCP Issues

Beyond the issues covered in previous topics (TIME_WAIT exhaustion, CLOSE_WAIT leaks, SYN floods), several TCP-related problems commonly affect production systems:

Connection Resets Behind Load Balancers

When a server closes an idle connection, the load balancer may not know immediately. If the LB sends a request on the stale connection, the server responds with RST ("Connection reset by peer"). The client sees a failed request.

This is the stale connection problem and it is extremely common in microservice architectures. Solutions include: using HTTP keep-alive with timeouts shorter than the server's idle timeout, implementing retry logic for connection reset errors (they are almost always transient), and configuring the load balancer's idle timeout to be shorter than the server's.

Slow Connections Due to Small Initial cwnd

A fresh TCP connection starts with a small congestion window (typically 10 MSS ≈ 14.6 KB). For API responses larger than this, the server must wait for ACKs before sending more data - each wait costs one RTT. A 50 KB API response on a 100ms RTT connection takes approximately 4 RTTs (400ms) just for the congestion window to grow large enough to send the complete response.

The fix is connection reuse: persistent connections (HTTP keep-alive) retain the expanded congestion window from previous requests. The first request on a new connection is always slower than subsequent requests - this is the "cold connection" penalty that connection pooling eliminates.

TCP Retransmission Storms

When a network path experiences packet loss, TCP retransmissions consume additional bandwidth. If the loss is caused by congestion (the link is overloaded), the retransmissions add more traffic to an already congested link, potentially making the congestion worse. This positive feedback loop can cause congestion collapse on severely overloaded links.

In production, retransmission storms typically manifest as sudden latency spikes and throughput drops that recover after several seconds (when congestion control reduces sending rates). Monitoring the retransmission rate (netstat -s | grep retransmit or Prometheus node_netstat_Tcp_RetransSegs) is one of the most useful network health indicators.

Connection Refused vs Connection Timeout

These two errors have completely different causes:

Connection refused (ECONNREFUSED): The server is reachable but nothing is listening on the requested port. The server kernel responds immediately with RST. This typically means the application is not running or is listening on a different port. Diagnosis is fast - the error is immediate.

Connection timeout (ETIMEDOUT): No response at all - not even RST. The client's SYN segments are silently dropped. This typically means a firewall is blocking traffic, the server is unreachable, or the server's SYN backlog is full. Diagnosis is slow - the client must wait for the SYN timeout (typically 1-3 seconds for the first attempt with exponential backoff for retries).

Distinguishing between these errors is one of the first steps in network troubleshooting. "Connection refused" means the network path works but the service is down. "Connection timeout" means the network path is broken or the service is overloaded at the connection level.

TCP Keepalive vs Application Keepalive

TCP keepalive (kernel-level, SO_KEEPALIVE) sends empty ACK segments to detect dead connections. Application keepalive (HTTP keep-alive, gRPC ping frames, WebSocket pings) sends application-level messages to maintain the connection and detect application-level failures.

Both serve similar purposes but operate at different layers. TCP keepalive detects network failures (the remote host crashed or became unreachable). Application keepalive detects application failures (the remote process hung, the thread pool is exhausted). In production, both should be configured:

• TCP keepalive: set tcp_keepalive_time shorter than any middlebox idle timeout (typically 60-300 seconds)
• Application keepalive: set based on the application's tolerance for stale connections (typically 15-60 seconds for gRPC, configurable for HTTP/2)

When TCP Is the Wrong Choice

TCP is the default choice for network communication, and it is the right choice for the vast majority of applications. But there are specific scenarios where TCP's guarantees create more problems than they solve:

Real-time media (video calls, live streaming). A dropped video frame is better than a delayed frame. TCP's retransmission delays cause visible stuttering as the player waits for retransmitted data instead of playing the next frame. RTP over UDP (or WebRTC's SRTP) is the standard for real-time media.

Online multiplayer gaming. Player position updates every 16ms (60 fps). A delayed position update from 200ms ago is useless - the player has moved since then. UDP allows the game to skip lost updates and process only the latest state, maintaining responsiveness even under packet loss.

DNS queries. DNS lookups are small (typically under 512 bytes), independent, and idempotent. The overhead of a TCP handshake (1 RTT) would double the latency of a DNS query. UDP eliminates the handshake, and the application retransmits if needed. (DNS over TCP is used for large responses and zone transfers.)

High-frequency sensor data / IoT telemetry. When sending thousands of small readings per second, the overhead of TCP connection management and per-packet acknowledgment is disproportionate. UDP with application-level batching and checksumming is more efficient.

Internal service mesh with controlled networks. Some high-performance internal systems (financial trading, HPC clusters) use RDMA or custom UDP-based protocols that bypass TCP's overhead entirely, sacrificing generality for microsecond-level latency.

graph TD
    subgraph Decision["When to Use TCP vs UDP vs QUIC"]
        direction TB
        Q1["Need reliability<br/>and ordering?"]
        Q1 -->|"Yes"| Q2["Multiplexing<br/>independent streams?"]
        Q2 -->|"Yes"| QUIC["QUIC / HTTP/3<br/>Independent stream<br/>reliability"]
        Q2 -->|"No"| TCP["TCP<br/>Standard reliable<br/>transport"]
        Q1 -->|"No - speed<br/>matters more"| Q3["Latency-critical<br/>real-time data?"]
        Q3 -->|"Yes"| UDP["UDP<br/>Minimal overhead<br/>App handles loss"]
        Q3 -->|"No"| TCP2["TCP is probably<br/>still fine"]
    end

    style Q1 fill:#4f46e5,stroke:#4338ca,color:#fff
    style Q2 fill:#7c3aed,stroke:#6d28d9,color:#fff
    style Q3 fill:#7c3aed,stroke:#6d28d9,color:#fff
    style TCP fill:#22c55e,stroke:#16a34a,color:#fff
    style QUIC fill:#22c55e,stroke:#16a34a,color:#fff
    style UDP fill:#f59e0b,stroke:#d97706,color:#fff
    style TCP2 fill:#22c55e,stroke:#16a34a,color:#fff

This decision tree shows when to choose TCP, UDP, or QUIC. First question: do you need reliable, ordered delivery? If yes, ask whether you're multiplexing independent streams - if so, QUIC provides independent stream reliability without head-of-line blocking; if not, TCP is the standard choice. If you don't need reliability and speed matters more, ask whether the data is latency-critical and real-time - if yes, UDP provides minimal overhead and the application handles loss. If not, TCP is probably still the right choice.

Module Summary

This module has taken you through TCP from first principles to production operations. You now understand the transport layer's role (process-to-process delivery via ports), TCP connection management (three-way handshake, four-way close, TIME_WAIT), TCP reliability (acknowledgments, retransmission, SACK), flow control (sliding window, BDP), congestion control (slow start, AIMD, CUBIC, BBR), TCP's fundamental limitations (HOL blocking), and practical tuning and troubleshooting. TCP carries the vast majority of internet traffic, and deep understanding of it is one of the highest-leverage skills for System Design interviews and production operations.

Interview: TCP tuning and operational knowledge demonstrates production experience:

"Your service has intermittent connection timeouts to a downstream API." - Check TCP keepalive settings. If the idle timeout of an intermediate load balancer or firewall is shorter than TCP keepalive (default 2 hours), idle connections are killed by the middlebox without notification. The next request on a dead connection times out. Solution: reduce tcp_keepalive_time to less than the middlebox timeout, or use application-level health checks.

"High P99 latency on a service with many short-lived connections." - Each new connection pays the TCP handshake cost (1 RTT) and starts in slow start. Enable connection pooling (HTTP keep-alive, database connection pools) to amortize connection setup across many requests. Also check TIME_WAIT accumulation - port exhaustion causes connection failures that spike latency.

"Should we use TCP or UDP for our real-time feature?" - If ordering and reliability are required (chat messages, financial transactions), use TCP. If timeliness matters more than completeness (live video, gaming state updates, real-time sensor data), use UDP or QUIC. The key question: is a delayed delivery worse than a skipped delivery?

Exercise: Starter: Examine your system's TCP parameters. Run sysctl -a | grep tcp (Linux) and identify the keepalive time, buffer sizes, congestion control algorithm, and SYN backlog. How do they compare to the production recommendations in this topic?

Intermediate: Create a TCP tuning profile for a high-throughput web server. Write a sysctl.conf snippet that sets: somaxconn to 4096, tcp_max_syn_backlog to 8192, tcp_keepalive_time to 120, tcp_tw_reuse to 1, ip_local_port_range to "1024 65535", and tcp_rmem/wmem max to 16 MB. Explain why each setting matters and what problem it prevents.

Advanced: Research TCP BBR deployment at Google scale. How did Google measure the impact of switching from CUBIC to BBR on YouTube traffic? What metrics improved (throughput, rebuffering, latency)? Why did some researchers raise concerns about BBR's fairness when competing with CUBIC flows? Read the original BBR paper (Cardwell et al., 2017) and summarize the key insights.

Back to Computer Networking