Free lesson · Computer Networking
Tcp Congestion Control
The Network's Speed Limit
Flow control prevents the sender from overwhelming the receiver. But what about the network itself? If every TCP sender transmitted at maximum speed simultaneously, routers between them would overflow their buffers, drop packets, and the network would collapse into congestion collapse - a state where the network is busy but almost no useful data gets through because most capacity is consumed by retransmissions of dropped packets.
This nearly happened in October 1986, when the early internet experienced its first congestion collapse. Throughput between UC Berkeley and LBL (separated by 400 yards of cable) dropped from 32 Kbps to 40 bps - a 99.9% reduction. Van Jacobson analyzed the problem and designed the TCP congestion control algorithms that all modern TCP implementations use. Without congestion control, the internet as we know it could not function.
Congestion control is a mechanism where each TCP sender independently estimates the network's available capacity and limits its transmission rate accordingly. Unlike flow control (where the receiver explicitly communicates its capacity via the window size), congestion control requires the sender to infer network capacity from indirect signals - primarily packet loss and, in newer algorithms, delay.
The Congestion Window (cwnd)
The sender maintains a congestion window (cwnd) - an internal estimate of how many bytes it can safely have in flight without causing congestion. The congestion window is not communicated in TCP headers; it exists only in the sender's memory. The effective transmission limit is:
Effective window = min(rwnd, cwnd)
The congestion window changes dynamically based on the sender's perception of network conditions. When the network seems healthy (ACKs arriving promptly), cwnd increases. When the network seems congested (packet loss detected), cwnd decreases sharply. This adaptive dynamic behavior is what keeps the network stable under varying load conditions.
The Four Phases of TCP Congestion Control
Classic TCP congestion control (Reno/NewReno) operates in four phases:
Phase 1 - Slow Start
Despite its name, slow start is actually fast - it is "slow" only compared to transmitting at full speed from the beginning. The sender starts with cwnd = 1 MSS (typically 1,460 bytes on Ethernet) and increases cwnd by 1 MSS for every ACK received. Because each ACK roughly corresponds to one MSS of data delivered, this doubles cwnd every round-trip time - exponential growth.
RTT 1: cwnd = 1 MSS → send 1 segment → receive 1 ACK → cwnd = 2
RTT 2: cwnd = 2 MSS → send 2 segments → receive 2 ACKs → cwnd = 4
RTT 3: cwnd = 4 MSS → send 4 segments → receive 4 ACKs → cwnd = 8
RTT 4: cwnd = 8 MSS → send 8 → cwnd = 16
Slow start continues until one of three things happens: cwnd reaches the slow start threshold (ssthresh), packet loss is detected, or cwnd reaches rwnd.
Modern Linux implementations start with an initial cwnd of 10 MSS (RFC 6928, approximately 14.6 KB on Ethernet) rather than 1 MSS, significantly improving performance for short connections like HTTP requests where the entire response might fit in the initial window. Google led the push for larger initial windows based on research showing that most web pages could be served within a 10 MSS initial window, eliminating multiple RTTs of slow start for typical page loads. Before this change, a single small web page could take 3-4 RTTs just to ramp up the congestion window - adding hundreds of milliseconds of latency on high-RTT connections.
Some CDN and cloud providers configure even larger initial windows (up to 32 or 64 MSS) for their edge servers, further reducing the impact of slow start on user-facing latency. This is possible because edge servers are close to users (low RTT) and connected by well-provisioned links where the risk of congestion from a larger initial burst is low.
Phase 2 - Congestion Avoidance
Once cwnd reaches ssthresh, TCP transitions from exponential growth (slow start) to linear growth (congestion avoidance). In congestion avoidance, cwnd increases by approximately 1 MSS per round-trip time, regardless of how many ACKs are received. This cautious growth probes for additional bandwidth without aggressively ramping up.
The formula: cwnd = cwnd + MSS × (MSS / cwnd) per ACK. This adds roughly 1 MSS per RTT - Additive Increase.
Phase 3 - Congestion Detection (Packet Loss)
TCP interprets packet loss as a signal of congestion. Loss is detected in two ways:
Timeout (severe congestion): The retransmission timer expires. TCP assumes this indicates severe congestion and responds drastically: ssthresh = cwnd / 2 and cwnd = 1 MSS. TCP returns to slow start. This is the Multiplicative Decrease half of AIMD.
Three duplicate ACKs (mild congestion - Fast Retransmit): Three duplicate ACKs indicate a single lost segment but the network is still delivering subsequent segments (they caused the duplicate ACKs). TCP treats this as mild congestion.
Phase 4 - Fast Recovery
After fast retransmit (triggered by three duplicate ACKs), TCP enters fast recovery instead of returning to slow start. In fast recovery: ssthresh = cwnd / 2 and cwnd = ssthresh + 3 MSS. The sender retransmits the lost segment and continues transmitting new data as duplicate ACKs arrive. When a non-duplicate ACK arrives (confirming the retransmitted segment was received), TCP exits fast recovery and enters congestion avoidance with cwnd = ssthresh.
Fast recovery avoids the performance cliff of returning to slow start. The logic is: three duplicate ACKs mean the network is still delivering some packets, so congestion is not severe enough to warrant starting over from scratch.
The combination of additive increase (congestion avoidance) and multiplicative decrease (loss response) is called AIMD (Additive Increase, Multiplicative Decrease). AIMD produces the characteristic "sawtooth" pattern in TCP throughput graphs - throughput gradually increases, drops sharply on loss detection, and gradually increases again.
graph LR
subgraph AIMD["TCP Congestion Control - AIMD Sawtooth"]
direction LR
SS["Slow Start<br/>(exponential growth)<br/>cwnd doubles<br/>each RTT"] -->|"cwnd reaches<br/>ssthresh"| CA["Congestion<br/>Avoidance<br/>(linear growth)<br/>+1 MSS per RTT"]
CA -->|"Packet loss<br/>(3 dup ACKs)"| FR["Fast Recovery<br/>cwnd = cwnd/2<br/>+ 3 MSS"]
FR -->|"New ACK"| CA
CA -->|"Timeout"| SS2["Back to<br/>Slow Start<br/>cwnd = 1 MSS"]
SS2 --> CA
end
style SS fill:#4f46e5,stroke:#4338ca,color:#fff
style CA fill:#7c3aed,stroke:#6d28d9,color:#fff
style FR fill:#8b5cf6,stroke:#7c3aed,color:#fff
style SS2 fill:#ef4444,stroke:#dc2626,color:#fff
This diagram shows the four phases of TCP congestion control in the AIMD model. Slow start begins with exponential growth, doubling the congestion window each round-trip time. When cwnd reaches the slow start threshold, TCP transitions to congestion avoidance with linear growth of one MSS per RTT. When packet loss is detected via three duplicate ACKs, TCP enters fast recovery, halving cwnd and adding 3 MSS. A new ACK returns TCP to congestion avoidance. If loss is detected via timeout (more severe), TCP returns to slow start with cwnd reset to 1 MSS.
Modern Congestion Control Algorithms
Classic Reno/NewReno congestion control uses packet loss as the sole congestion signal. This works but has significant limitations: loss-based algorithms fill router buffers to capacity before detecting congestion (causing bufferbloat - high latency even when throughput is adequate), and they perform poorly on high-bandwidth, high-latency links where a single loss event triggers a massive cwnd reduction.
Modern algorithms address these limitations:
CUBIC (Linux default since 2006) modifies the congestion window growth function to be more aggressive in reclaiming bandwidth after a loss event. Instead of linear growth, CUBIC uses a cubic function that grows slowly near the previous maximum window (probing cautiously) and faster when far from it (recovering bandwidth quickly). CUBIC performs significantly better than Reno on high-BDP connections.
BBR (Bottleneck Bandwidth and Round-trip propagation time) was developed by Google and takes a fundamentally different approach. Instead of using packet loss as a congestion signal, BBR continuously estimates the connection's bottleneck bandwidth and minimum RTT, and targets a sending rate that matches the bottleneck bandwidth without filling the buffers. BBR reduces bufferbloat (maintaining low latency) and achieves higher throughput than loss-based algorithms on links with random (non-congestion) packet loss. BBR is used on Google's internal network and YouTube servers.
QUIC (the transport protocol behind HTTP/3) implements congestion control at the application layer rather than in the kernel, allowing faster experimentation and deployment of new algorithms. QUIC initially used CUBIC and has progressively adopted BBR variants.
Note: Founder Callout - Nikhil (Google): Google Spanner's cross-region replication pushes TCP congestion control to its limits. Replicating data between data centers on different continents means high-bandwidth links with 100-200ms RTT - classic Long Fat Network conditions. A single packet loss with classic Reno congestion control could halve throughput and take seconds to recover. Google's deployment of BBR internally was partly motivated by Spanner's need for consistent, high-throughput, low-latency replication across these intercontinental links. When you design a multi-region system and specify "strong consistency with Spanner-like replication," you are implicitly depending on the congestion control algorithm being smart enough to handle those link characteristics.
Explicit Congestion Notification (ECN)
Traditional congestion control relies on packet loss to signal congestion - but by the time packets are dropped, the network is already overloaded. ECN (Explicit Congestion Notification) allows routers to signal congestion before dropping packets by marking packets with a congestion indication in the IP header.
When ECN is enabled (negotiated during the TCP handshake), a router experiencing congestion sets the CE (Congestion Experienced) bits in the IP header instead of dropping the packet. The receiver sees the CE marking and sets the ECE (ECN-Echo) flag in its next TCP ACK. The sender receives the ECE flag and reduces cwnd as if a loss had occurred - but no data was actually lost, so no retransmission is needed.
ECN improves performance by triggering congestion response earlier (before buffer overflow) and without the retransmission delay that accompanies packet loss. It is particularly beneficial for latency-sensitive applications. ECN support is available in all modern operating systems but is not always enabled by default - on Linux, it can be enabled with sysctl -w net.ipv4.tcp_ecn=1.
Congestion Control and Fairness
When multiple TCP flows share a bottleneck link, congestion control must ensure fairness - each flow should get an approximately equal share of the available bandwidth. AIMD achieves this mathematically: when all flows reduce their windows by half on congestion and increase linearly otherwise, the system converges toward equal sharing regardless of starting conditions.
However, fairness is complicated by several factors:
RTT unfairness. Flows with shorter RTTs increase cwnd faster (they complete more round trips per unit time) and claim a larger share of bandwidth. A flow with 10ms RTT grows 10x faster than a flow with 100ms RTT. This RTT bias is inherent in TCP and means that geographically closer connections get better performance - a well-known issue for global services.
Algorithm heterogeneity. When BBR flows share a link with CUBIC flows, the two algorithms may not share bandwidth equally. BBR can claim more than its fair share in some scenarios, which has generated debate in the networking research community about deployment implications.
Application behavior. A single application opening many parallel TCP connections (as HTTP/1.1 browsers do, typically 6 per domain) claims a proportionally larger share of the bottleneck bandwidth. This is a form of "unfairness by parallelism" that HTTP/2 addresses by multiplexing multiple requests over a single connection.
Why Congestion Control Matters for Application Performance
Congestion control directly affects the throughput and latency your application experiences:
New connections start slow. A fresh TCP connection begins in slow start with a small cwnd. On a 100ms RTT connection, it takes several hundred milliseconds to ramp up to full throughput. Short-lived HTTP connections (common in REST APIs) may complete their request-response cycle before slow start finishes, never achieving full bandwidth. This is why connection pooling, HTTP/2 multiplexing, and persistent connections dramatically improve performance - they amortize the slow start cost across many requests.
Packet loss causes throughput drops. Even a single lost packet triggers a cwnd reduction. On a high-bandwidth link, halving cwnd can reduce throughput by hundreds of Mbps. Recovery takes multiple RTTs of linear growth. The practical impact: intermittent packet loss (even at 0.1%) causes noticeable throughput variability, especially on long-distance links.
Bufferbloat adds latency. Loss-based congestion control fills buffers before detecting congestion. If a router has a large buffer, TCP fills it completely before losing a packet, adding hundreds of milliseconds of queuing delay. BBR mitigates this by targeting a sending rate that avoids filling buffers entirely.
graph TD
subgraph Algorithms["Congestion Control Algorithm Evolution"]
direction TB
RENO["Reno / NewReno<br/>Loss-based<br/>AIMD<br/>(1988-1999)"]
CUBIC["CUBIC<br/>Loss-based<br/>Cubic function growth<br/>Linux default (2006+)"]
BBR["BBR<br/>Model-based<br/>Bandwidth + RTT estimation<br/>Google (2016+)"]
RENO --> CUBIC --> BBR
end
style RENO fill:#4f46e5,stroke:#4338ca,color:#fff
style CUBIC fill:#7c3aed,stroke:#6d28d9,color:#fff
style BBR fill:#22c55e,stroke:#16a34a,color:#fff
This diagram shows the evolution of TCP congestion control algorithms. Reno and NewReno, designed in 1988 to 1999, are loss-based using the AIMD approach. CUBIC, the Linux default since 2006, improves on Reno with a cubic growth function that recovers bandwidth faster after loss. BBR, developed by Google in 2016, takes a fundamentally different model-based approach, estimating bottleneck bandwidth and round-trip time instead of using loss as the congestion signal.
Interview: Congestion control is a frequent System Design topic:
"Why is the first page load slow but subsequent loads fast?" - The first request uses a cold TCP connection in slow start with a small cwnd. Subsequent requests reuse the warmed-up connection (via HTTP keep-alive or HTTP/2) with an already-expanded cwnd. Connection reuse is one of the highest-impact performance optimizations.
"How does TCP affect throughput between regions?" - The BDP determines the minimum window needed to fill the pipe. Congestion control's slow start and AIMD sawtooth limit actual throughput. Packet loss on long-distance links has an outsized impact because cwnd reduction is multiplicative and recovery is linear. BBR mitigates this by not using loss as the primary congestion signal.
"What is bufferbloat?" - Loss-based congestion control fills router buffers before detecting congestion. Large buffers (common in modern routers) add hundreds of milliseconds of queuing delay before a loss event triggers cwnd reduction. BBR and Active Queue Management (AQM) techniques like CoDel and FQ-CoDel address bufferbloat.
What Comes Next
The next topic covers TCP connection termination - the four-way close process and the critically important TIME_WAIT state that prevents old segments from interfering with new connections.
Exercise: Starter: Check which congestion control algorithm your system uses. On Linux:
sysctl net.ipv4.tcp_congestion_control. Common values arecubic(default) orbbr. List available algorithms:sysctl net.ipv4.tcp_available_congestion_control.Intermediate: Use
iperf3to measure TCP throughput between two machines (or between your machine and a public iperf3 server). Observe the throughput ramp-up during slow start. Then switch the congestion control algorithm (sysctl -w net.ipv4.tcp_congestion_control=bbr) and re-run the test. Is there a measurable difference in throughput or latency? On high-latency connections, the difference should be significant.Advanced: Research the fairness properties of AIMD. Why does AIMD converge to equal bandwidth sharing when multiple TCP flows share a bottleneck link? Draw the phase plot (sender 1 throughput vs sender 2 throughput) and trace how AIMD moves the operating point toward the fairness line. Then research how BBR's model-based approach interacts with CUBIC flows on the same bottleneck - does BBR achieve more or less than its fair share? (This is an active area of research and debate.)