Free lesson · Computer Networking

Tcp Three Way Handshake

Easy10 min readnetworkingcomputer-networkstcp-ip

Why Connections Need a Handshake

TCP is a connection-oriented protocol. Before any data can flow between a client and server, both sides must agree that a connection exists, synchronize their sequence numbers, and negotiate parameters. This agreement is established through the three-way handshake - a sequence of three messages (SYN, SYN-ACK, ACK) that creates a TCP connection in one round trip.

The handshake is not just a formality. It serves three critical purposes: it verifies that both endpoints are reachable and willing to communicate, it synchronizes the initial sequence numbers that both sides will use to track data, and it negotiates connection parameters like the maximum segment size (MSS) and window scaling factor. Without the handshake, TCP could not provide its reliability and ordering guarantees - sequence numbers would be unsynchronized, and the connection state would be undefined.

Every HTTPS request your browser makes, every database query your application sends, every SSH session you open begins with a three-way handshake. Understanding it explains why TCP connections have latency overhead (at minimum, one round trip before data can flow), and why protocols like QUIC were designed to eliminate or reduce this overhead.

The Three Steps

Step 1 - SYN (Client → Server)

The client initiates the connection by sending a TCP segment with the SYN (Synchronize) flag set. This segment contains:

The SYN flag set to 1
The client's Initial Sequence Number (ISN) - a randomly chosen 32-bit number that will be the starting point for numbering the client's data bytes
The client's MSS (Maximum Segment Size) option - the largest data payload the client can receive in a single TCP segment
Optionally, window scaling and other TCP options

The client transitions from the CLOSED state to the SYN-SENT state and waits for a response.

Step 2 - SYN-ACK (Server → Client)

The server receives the SYN, allocates resources for the connection, and responds with a segment that has both the SYN and ACK flags set:

The SYN flag set to 1 (the server is synchronizing its own sequence number)
The server's own ISN - a different randomly chosen 32-bit number
The ACK flag set to 1, with the acknowledgment number set to the client's ISN + 1 (acknowledging receipt of the client's SYN)
The server's MSS and other options

The server transitions from LISTEN to the SYN-RECEIVED state.

Step 3 - ACK (Client → Server)

The client receives the SYN-ACK, confirms the server's sequence number, and sends a final ACK:

The ACK flag set to 1, with the acknowledgment number set to the server's ISN + 1
The SYN flag is NOT set - this is a pure acknowledgment
This segment can also carry data (though it often does not)

Both sides transition to the ESTABLISHED state. The connection is now open, and data can flow in both directions.

sequenceDiagram
    participant C as Client
    participant S as Server

    Note over S: State: LISTEN
    Note over C: State: CLOSED

    C->>S: SYN<br/>Seq=100 (client ISN)<br/>MSS=1460
    Note over C: State: SYN-SENT

    S->>C: SYN-ACK<br/>Seq=300 (server ISN)<br/>Ack=101<br/>MSS=1460
    Note over S: State: SYN-RECEIVED

    C->>S: ACK<br/>Seq=101<br/>Ack=301
    Note over C: State: ESTABLISHED
    Note over S: State: ESTABLISHED

    Note over C,S: Connection established<br/>Data can now flow

This sequence diagram shows the TCP three-way handshake. The server starts in the LISTEN state and the client starts in CLOSED. The client sends a SYN segment with sequence number 100 as its initial sequence number and MSS of 1460 bytes, transitioning to the SYN-SENT state. The server responds with a SYN-ACK, using sequence number 300 as its own initial sequence number and acknowledgment number 101, confirming receipt of the client's SYN. The server moves to SYN-RECEIVED. The client sends a final ACK with sequence number 101 and acknowledgment 301. Both sides enter the ESTABLISHED state and data can flow.

Why Three Steps, Not Two?

A natural question: why does TCP need three messages to establish a connection? Why not just two (SYN and SYN-ACK)?

The answer is that both sides need to synchronize sequence numbers, and both sides need to know that the other has received their sequence number. Two messages would be sufficient for one-way synchronization - the client sends a SYN, the server acknowledges it in the SYN-ACK - but the server's sequence number has not been acknowledged. Without the third ACK, the server does not know if the client received its ISN. The server would be sending data using sequence numbers that the client might not have received - making reliable data transfer impossible.

Additionally, the two-way handshake is vulnerable to a specific failure: duplicate SYN segments. If the client sends a SYN that gets delayed in the network, and then sends another SYN (because it timed out), the server might receive both SYNs and establish two connections when only one was intended. The third step (ACK) allows the server to verify that the client is still actively trying to connect - a delayed, duplicate SYN from an earlier attempt would not be followed by a valid ACK, so the server can discard it.

Initial Sequence Numbers (ISNs)

ISNs are not zero. They are randomly chosen 32-bit numbers, different for each connection. This randomness serves two purposes:

Preventing confusion from old connections. If ISNs always started at zero, segments from a previous connection between the same IP addresses and ports could be mistakenly accepted as part of a new connection. Random ISNs make this collision astronomically unlikely - the segments from the old connection would have sequence numbers in an entirely different range than the new connection.

Security. Predictable ISNs were historically exploited in TCP sequence prediction attacks, where an attacker guessed the server's ISN and injected spoofed packets into a connection. The famous Mitnick attack of 1994 exploited predictable ISNs. Modern operating systems use cryptographically random ISN generation (RFC 6528) to prevent this class of attack.

MSS Negotiation

During the handshake, both sides advertise their Maximum Segment Size (MSS) - the largest TCP payload they can receive. MSS is communicated as a TCP option in the SYN and SYN-ACK segments. The MSS is calculated from the local interface's MTU minus the IP header (20 bytes) and TCP header (20 bytes):

MSS = MTU - IP header - TCP header
MSS = 1500 - 20 - 20 = 1460 bytes (typical for Ethernet)

Each side independently advertises its MSS, and both sides use the smaller of the two values to size their outgoing segments. If your machine advertises MSS 1460 and the server advertises MSS 1380 (perhaps because it is behind a VPN with reduced MTU), both sides limit their segments to 1,380 bytes.

MSS is not negotiated - it is announced. Each side states what it can receive, and the other side respects that limit. This prevents fragmentation at the IP layer by ensuring TCP segments are sized to fit within the path's MTU from the start.

Window Scaling

The TCP header's Window Size field is 16 bits, giving a maximum receive window of 65,535 bytes. For high-bandwidth, high-latency connections (like intercontinental links), this is far too small - the sender would have to stop and wait for acknowledgments before the window fills, dramatically underutilizing the available bandwidth.

Window scaling (RFC 7323) is a TCP option negotiated during the handshake. Both sides include a window scale factor in their SYN and SYN-ACK. The scale factor is a power of 2 (0-14) that is applied to all subsequent Window Size values. A scale factor of 7 means each window size value is multiplied by 128, allowing windows up to 65,535 × 128 = 8,388,480 bytes (~8 MB). This is essential for modern high-speed networks.

Window scaling must be agreed upon during the handshake - it cannot be added later. If either side does not include the window scale option, window scaling is disabled for the entire connection.

Other TCP Options Negotiated in the Handshake

SACK Permitted: Selective Acknowledgment allows the receiver to acknowledge non-contiguous data, improving retransmission efficiency. Both sides must agree to SACK during the handshake.

Timestamps: TCP timestamps (RFC 7323) serve two purposes: they improve RTT measurement accuracy (essential for computing retransmission timeouts) and they protect against sequence number wraparound on very high-speed connections (PAWS - Protection Against Wrapped Sequences). Like window scaling, timestamps are negotiated during the handshake.

Simultaneous Open

In rare cases, both sides can send SYN simultaneously - each tries to initiate a connection to the other at the same time. This is called simultaneous open. TCP handles it correctly: each side receives the other's SYN, both respond with SYN-ACK, and both transition to ESTABLISHED. The result is a single connection, not two. While simultaneous open is uncommon in practice (most connections follow the client-server model), the TCP specification explicitly supports it for completeness.

The TCP Connection State Machine

TCP connections follow a well-defined state machine. The three-way handshake involves these states:

State	Meaning
CLOSED	No connection exists
LISTEN	Server is waiting for incoming SYN
SYN-SENT	Client has sent SYN, waiting for SYN-ACK
SYN-RECEIVED	Server has received SYN and sent SYN-ACK, waiting for ACK
ESTABLISHED	Connection is fully open, data can flow

Understanding TCP states is essential for debugging. When you run netstat or ss and see connections in various states, the state tells you exactly where in the connection lifecycle each connection is. A server with thousands of connections in SYN-RECEIVED might be under a SYN flood attack. A server with connections stuck in ESTABLISHED with no data flow might have application-level hangs. The state machine is the diagnostic framework.

The Listen Backlog

When a server calls listen(backlog), the backlog parameter controls how many connections can be in the handshake process simultaneously. The kernel maintains two queues:

SYN queue (half-open connections): Connections that have received a SYN and sent SYN-ACK but not yet received the final ACK. These are in the SYN-RECEIVED state.

Accept queue (completed connections): Connections that have completed the three-way handshake (ESTABLISHED) but have not yet been accepted by the application via accept().

If the SYN queue fills up (too many half-open connections), the server drops new SYN segments - which is exactly how SYN flood attacks work. The attacker fills the SYN queue with half-open connections that will never complete, preventing legitimate connections from being established.

If the accept queue fills up (the application is not calling accept() fast enough), the kernel may either drop new completed connections or send RST to reject them, depending on the tcp_abort_on_overflow setting. A full accept queue is a common cause of connection refused errors on overloaded servers - the server is technically reachable, but the application cannot keep up with incoming connections.

Handshake Latency and Performance

The three-way handshake introduces exactly one round-trip time (RTT) of latency before any data can be sent. If the RTT between client and server is 50ms, the handshake alone takes 50ms. This is in addition to any DNS resolution time and TLS handshake time.

For HTTPS connections (which add a TLS handshake on top of the TCP handshake), the total connection establishment can take 2-3 round trips:

TCP handshake: 1 RTT
TLS 1.2 handshake: 2 RTTs (TLS 1.3 reduces this to 1 RTT)
First HTTP request: 1 additional RTT

On a 100ms RTT connection (common for cross-continent communication), this means 200-300ms before the first byte of application data arrives. This latency is one of the primary motivations behind several modern protocol optimizations:

TCP Fast Open (TFO): Allows data to be sent in the SYN segment itself, eliminating one RTT for repeat connections. The server issues a cookie during the first connection that the client includes in subsequent SYN segments, allowing the server to validate the client and accept data immediately.

TLS 1.3: Reduces the TLS handshake from 2 RTTs to 1 RTT, and supports 0-RTT resumption for repeat connections to the same server.

QUIC (HTTP/3): Combines the transport handshake and cryptographic handshake into a single round trip, achieving 1 RTT for new connections and 0 RTT for resumed connections.

graph LR
    subgraph Latency["Connection Establishment Latency"]
        direction LR
        TCP["TCP Handshake<br/>1 RTT"] --> TLS["TLS 1.3<br/>Handshake<br/>1 RTT"] --> HTTP["First HTTP<br/>Request<br/>1 RTT"]
    end

    subgraph QUIC_Path["QUIC (HTTP/3)"]
        direction LR
        QUIC["Combined Transport<br/>+ Crypto Handshake<br/>1 RTT"] --> QUIC_HTTP["First HTTP<br/>Request<br/>Same RTT"]
    end

    style TCP fill:#4f46e5,stroke:#4338ca,color:#fff
    style TLS fill:#7c3aed,stroke:#6d28d9,color:#fff
    style HTTP fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style QUIC fill:#22c55e,stroke:#16a34a,color:#fff
    style QUIC_HTTP fill:#22c55e,stroke:#16a34a,color:#fff

This diagram compares connection establishment latency. TCP plus TLS 1.3 requires 2 round trips before data can flow: 1 RTT for the TCP handshake and 1 RTT for the TLS handshake, followed by the first HTTP request. QUIC combines the transport and cryptographic handshakes into a single round trip, achieving 1 RTT total before data flows. For resumed connections, QUIC can achieve 0 RTT.

Interview: The three-way handshake is one of the most commonly asked networking questions:

"Walk me through the TCP handshake." - SYN (client sends initial sequence number), SYN-ACK (server acknowledges and sends its own ISN), ACK (client acknowledges server's ISN). Both sides enter ESTABLISHED state. It takes one RTT.

"Why three steps, not two?" - Both sides need to synchronize sequence numbers. Two messages would leave the server's ISN unacknowledged. The third step also protects against duplicate SYNs from old connections creating ghost connections.

"What is a SYN flood?" - An attacker sends many SYN segments from spoofed IP addresses. The server allocates resources (memory for connection state) for each SYN-RECEIVED connection and waits for ACKs that never come. The server's connection table fills up, and legitimate connections are rejected. SYN cookies mitigate this by encoding connection state in the SYN-ACK's sequence number rather than allocating server memory.

"How does TCP Fast Open reduce latency?" - TFO allows data to be sent in the SYN segment on repeat connections. The server validates the client using a cookie from a previous connection, eliminating one RTT for the handshake.

What Comes Next

The next topic covers TCP reliable data transfer - how TCP guarantees that every byte sent by the sender is received by the receiver, in order, exactly once (or at least once, in reality). This involves acknowledgments, retransmission timers, duplicate detection, and the delicate balance between reliability and performance.

Exercise: Starter: Capture a TCP handshake using a command-line tool. On Linux/Mac: sudo tcpdump -i any -c 3 port 443 -S and then in another terminal: curl https://google.com. You should see three packets: a SYN, a SYN-ACK, and an ACK. Identify the initial sequence numbers in each.

Intermediate: Use Wireshark (a graphical packet analyzer) to capture a TCP connection. Filter by tcp.flags.syn == 1 to find handshakes. Examine the TCP options in the SYN and SYN-ACK: MSS, window scaling, SACK permitted, timestamps. What MSS does your machine advertise? What window scaling factor does it request?

Advanced: Research SYN cookies (RFC 4987) in detail. How does a server encode connection state (ISN, MSS, window scale) into the SYN-ACK's sequence number? What information is lost when using SYN cookies (hint: TCP options from the SYN are not stored)? Why is this an acceptable trade-off under attack? How does the Linux kernel implementation (net.ipv4.tcp_syncookies) automatically enable SYN cookies only when the SYN backlog is full?

Back to Computer Networking