Free lesson · Computer Networking
Ip Fragmentation And Mtu
Why Packets Have a Size Limit
Every network link has a maximum frame size it can carry. An Ethernet link can carry frames up to 1,518 bytes (1,500 bytes of payload plus 18 bytes of Ethernet header). A Wi-Fi link may support different maximums. An old serial link might handle only 576 bytes. The MTU (Maximum Transmission Unit) is the largest IP packet (in bytes) that a link can carry without being broken into smaller pieces.
The standard Ethernet MTU is 1,500 bytes. This is the most common MTU on the internet and the default on virtually every network interface you will encounter. An IP packet up to 1,500 bytes fits in a single Ethernet frame and travels the link intact. An IP packet larger than 1,500 bytes does not fit - and must be either fragmented into smaller pieces or dropped.
This size constraint creates a fundamental tension in networking: applications want to send data in whatever size is convenient, but the network infrastructure imposes physical limits on packet size. IP fragmentation is the mechanism that resolves this tension - and Path MTU Discovery is the optimization that mostly eliminates the need for it.
How IP Fragmentation Works
When a router receives a packet that is larger than the MTU of the outgoing link, it has two options: fragment the packet into smaller pieces that each fit within the MTU, or drop the packet and send an ICMP error back to the sender. Which option the router chooses depends on the Don't Fragment (DF) flag in the IP header.
Fragmentation Process (DF flag not set)
If the DF flag is not set, the router fragments the packet:
- The router divides the packet's payload into fragments, each small enough to fit within the outgoing MTU (accounting for the IP header overhead that each fragment needs)
- Each fragment gets its own IP header with the same source and destination IP addresses as the original
- The Identification field is set to the same value in all fragments - this tells the destination that these fragments belong together
- The Fragment Offset field indicates where in the original packet each fragment's data belongs
- The More Fragments (MF) flag is set on all fragments except the last one, signaling that more fragments follow
The fragments travel independently through the network - they may take different paths and arrive out of order. The destination host is responsible for reassembling the fragments into the complete original packet using the Identification field, Fragment Offset, and MF flag.
Fragmentation with the DF Flag Set
If the DF (Don't Fragment) flag is set and the packet exceeds the outgoing MTU, the router drops the packet and sends an ICMP "Fragmentation Needed" (Type 3, Code 4) message back to the source. This message includes the MTU of the link where the packet was too large, enabling the source to reduce its packet size accordingly.
Most modern systems set the DF flag on all packets. This prevents fragmentation at intermediate routers and instead relies on Path MTU Discovery to determine the correct packet size before sending.
graph LR
subgraph Fragmentation["IP Fragmentation Example"]
direction LR
ORIG["Original Packet<br/>3,000 bytes<br/>ID: 12345"] --> R["Router<br/>Outgoing MTU:<br/>1,500 bytes"]
R --> F1["Fragment 1<br/>1,500 bytes<br/>ID: 12345<br/>Offset: 0<br/>MF: 1"]
R --> F2["Fragment 2<br/>1,500 bytes<br/>ID: 12345<br/>Offset: 185<br/>MF: 0"]
end
style ORIG fill:#4f46e5,stroke:#4338ca,color:#fff
style R fill:#7c3aed,stroke:#6d28d9,color:#fff
style F1 fill:#8b5cf6,stroke:#7c3aed,color:#fff
style F2 fill:#a78bfa,stroke:#8b5cf6,color:#fff
This diagram shows IP fragmentation in action. An original packet of 3,000 bytes with identification number 12345 arrives at a router whose outgoing link has an MTU of 1,500 bytes. The router splits the packet into two fragments. Fragment 1 is 1,500 bytes with offset 0 and the More Fragments flag set to 1, indicating more fragments follow. Fragment 2 is 1,500 bytes with a non-zero offset and More Fragments set to 0, indicating it is the last fragment. Both fragments carry the same identification number 12345 so the destination can reassemble them.
Why Fragmentation Is Bad
Fragmentation works, but it introduces significant problems that make it undesirable in practice:
Performance overhead. Each fragment requires its own IP header (20 bytes minimum), wasting bandwidth. A 3,000-byte packet split into two 1,500-byte fragments wastes 20 bytes on the second IP header. At scale, this overhead is non-trivial.
Reassembly burden on the destination. The destination host must buffer all fragments and reassemble them. This requires memory, CPU, and timeout management. If one fragment is lost, the entire original packet must be retransmitted - the destination cannot use the other fragments, and they are wasted. Lost fragments are common because network congestion affects fragments independently.
Head-of-line blocking. If fragments arrive out of order, the destination must wait for missing fragments before delivering the reassembled packet to the application. This adds latency that would not exist if the packet were sent at the correct size in the first place.
Security vulnerabilities. Fragmentation has been exploited in numerous network attacks. The Ping of Death attack sent malformed fragments that, when reassembled, created a packet larger than the maximum allowed size, causing buffer overflows on vulnerable systems. Teardrop attacks sent overlapping fragments that confused reassembly logic, causing crashes. Tiny fragment attacks used very small fragments to split TCP headers across multiple fragments, bypassing firewall rules that could only inspect complete headers.
Firewall complications. Firewalls that inspect packet contents (deep packet inspection) struggle with fragments. The first fragment contains the TCP/UDP header and can be inspected normally. But subsequent fragments contain only payload data without transport headers - the firewall cannot determine the source/destination port or whether the packet matches a firewall rule. Many firewalls simply drop non-first fragments, which effectively breaks fragmented traffic.
IPv6 and Fragmentation
IPv6 takes a stronger stance against fragmentation than IPv4. In IPv6, routers are not allowed to fragment packets - only the sending host can perform fragmentation. If an IPv6 router receives a packet that exceeds the outgoing link's MTU, it drops the packet and sends an ICMPv6 "Packet Too Big" message back to the sender. This design forces the sender to use Path MTU Discovery and send correctly sized packets from the start.
Additionally, IPv6 mandates a minimum MTU of 1,280 bytes on all links (compared to IPv4's minimum of 576 bytes). This higher minimum ensures that even the smallest IPv6 link can carry reasonably sized packets.
IPv6 fragmentation, when it occurs, is handled through an extension header rather than fields in the main IP header. This keeps the fixed IPv6 header simple (40 bytes, no fragmentation fields) and only adds overhead when fragmentation is actually needed - which should be rare if PMTUD is working correctly.
Reassembly in Detail
When the destination host receives fragments, it must reassemble them into the original packet. The reassembly process involves several challenges:
Buffer management. The destination must allocate buffer space for the incoming fragments and hold them until all fragments arrive. The reassembly buffer has a timeout (typically 60 seconds in Linux) - if all fragments do not arrive within this window, the entire set is discarded and an ICMP "Fragment Reassembly Time Exceeded" message is sent back to the sender.
Memory exhaustion risk. An attacker can exploit reassembly by sending many fragments for different packets, consuming the target's reassembly buffer memory. This is the basis of fragmentation-based denial-of-service attacks. Modern operating systems limit reassembly buffer sizes and the number of concurrent reassembly operations to mitigate this risk.
For all these reasons, modern networking aggressively avoids fragmentation. The standard approach is Path MTU Discovery.
Path MTU Discovery (PMTUD)
Path MTU Discovery (PMTUD), defined in RFC 1191, is the mechanism that allows a sender to determine the smallest MTU along the entire path to a destination - the Path MTU - and send packets that fit within it. This eliminates the need for fragmentation entirely.
How PMTUD Works
- The sender sets the Don't Fragment (DF) flag on all outgoing packets and sends them at the local interface's MTU (typically 1,500 bytes)
- If a router along the path has a smaller MTU, it drops the packet (because DF is set) and sends an ICMP "Fragmentation Needed" message back to the sender, including the MTU of the bottleneck link
- The sender receives the ICMP message and reduces its packet size to fit the reported MTU
- The sender retransmits with the smaller packet size
- This process repeats until the packet successfully reaches the destination, at which point the sender has discovered the path MTU
In practice, the path MTU is almost always 1,500 bytes (standard Ethernet MTU), because the vast majority of the internet is built on Ethernet links. Exceptions include VPN tunnels (which add encapsulation headers, reducing the effective MTU to 1,400-1,460 bytes depending on the VPN protocol), some legacy or specialized links, and paths that cross PPPoE connections (common in DSL and some fiber deployments, where PPPoE headers reduce the MTU to 1,492 bytes).
PMTUD Black Holes
PMTUD has a critical dependency: it requires ICMP "Fragmentation Needed" messages to reach the sender. If any firewall or router along the path blocks ICMP, the sender never learns that its packets are too large. The oversized packets are silently dropped, and the sender keeps retransmitting at the same size - creating a PMTUD black hole where the connection stalls.
PMTUD black holes are one of the most frustrating networking problems to diagnose. The symptoms look like a connection that establishes successfully (the small TCP SYN/SYN-ACK packets fit within any MTU) but stalls when transferring data (the larger data packets exceed the path MTU and are dropped without ICMP notification). This is why blocking ICMP indiscriminately is dangerous - it can break PMTUD and cause mysterious connection failures.
MSS clamping is a common workaround for PMTUD black holes. The router or firewall at the tunnel endpoint modifies the TCP MSS (Maximum Segment Size) option in the TCP handshake, advertising a smaller MSS that accounts for the tunnel overhead. This causes the sender to use smaller TCP segments from the start, avoiding the need for PMTUD. MSS clamping is standard practice on VPN gateways and PPPoE connections.
Jumbo Frames
While the standard Ethernet MTU is 1,500 bytes, many data center networks support jumbo frames with an MTU of up to 9,000 bytes (sometimes 9,216 bytes). Jumbo frames reduce per-packet overhead - a 9,000-byte frame carries 6x more data per IP header than a 1,500-byte frame - which improves throughput for bulk data transfers.
Jumbo frames are commonly used in data center environments for storage traffic (iSCSI, NFS), inter-server communication within a cluster, and high-performance computing workloads. However, every device along the path must support jumbo frames - a single device with a 1,500-byte MTU forces fragmentation or PMTUD reduction for the entire path. This makes jumbo frames practical only in controlled environments where every switch, router, and server is configured consistently.
Cloud providers typically support jumbo frames within their networks - AWS supports 9,001-byte MTU between instances in the same VPC on most instance types - but traffic leaving the VPC toward the internet uses the standard 1,500-byte MTU.
graph TD
subgraph PMTUD["Path MTU Discovery Process"]
direction TB
S1["Sender sends packet<br/>1,500 bytes, DF=1"] --> R1["Router 1<br/>MTU: 1,500<br/>✅ Forwards"]
R1 --> R2["Router 2<br/>MTU: 1,400 (VPN tunnel)<br/>❌ Drops packet"]
R2 -->|"ICMP: Fragmentation Needed<br/>Next-hop MTU: 1,400"| S1
S1 -->|"Retransmit at 1,400 bytes"| R1
R1 --> R2
R2 -->|"✅ Forwards"| DST["Destination"]
end
style S1 fill:#4f46e5,stroke:#4338ca,color:#fff
style R1 fill:#22c55e,stroke:#16a34a,color:#fff
style R2 fill:#ef4444,stroke:#dc2626,color:#fff
style DST fill:#22c55e,stroke:#16a34a,color:#fff
This diagram illustrates Path MTU Discovery. The sender transmits a 1,500-byte packet with the Don't Fragment flag set. Router 1 with a 1,500-byte MTU forwards it successfully. Router 2 has a 1,400-byte MTU due to a VPN tunnel and drops the packet because it is too large and fragmentation is forbidden. Router 2 sends an ICMP Fragmentation Needed message back to the sender, reporting the 1,400-byte MTU. The sender retransmits at 1,400 bytes, and the packet successfully passes through both routers to reach the destination.
MTU in Practice
Understanding MTU is essential for troubleshooting a common class of production issues:
VPN tunnels reduce effective MTU. IPsec adds 50-70 bytes of overhead. GRE adds 24 bytes. VXLAN adds 50 bytes. A packet that fits a 1,500-byte MTU before encapsulation may not fit after. If PMTUD is working, the sender adjusts. If PMTUD is broken (ICMP blocked), connections stall on large transfers.
Docker and Kubernetes networking involves multiple layers of encapsulation (VXLAN, IPinIP, or other overlay protocols), each reducing the effective MTU. Misconfigured MTU in container networking is a common cause of intermittent connectivity issues - small packets (API calls, health checks) work fine, but large transfers (file uploads, database queries returning large result sets) fail or stall.
Cloud inter-region traffic may traverse VPN tunnels or encapsulated links with reduced MTU. Most cloud providers handle this transparently with MSS clamping, but custom VPN configurations can create PMTUD issues if not configured carefully.
Interview: MTU and fragmentation appear in production debugging scenarios:
"TCP connections establish but data transfer stalls." - Classic PMTUD black hole. The TCP handshake uses small packets (SYN/SYN-ACK are typically 60-80 bytes) that fit any MTU. Large data packets exceed the path MTU, are dropped, and no ICMP reaches the sender. Solution: enable MSS clamping on the tunnel endpoint or fix the ICMP filtering.
"Why is our VPN throughput lower than expected?" - The tunnel encapsulation reduces the effective MTU. If the application sends 1,500-byte packets, each one triggers fragmentation or PMTUD reduction, adding overhead. Configuring the tunnel interface MTU correctly eliminates this issue.
"Small API calls work, large responses fail." - The small requests fit within the path MTU. Large responses exceed it. This pattern strongly suggests an MTU mismatch somewhere in the path, often in a container networking overlay.
What Comes Next
The final topic in this module covers NAT in the Routing Context - examining how Network Address Translation interacts with routing decisions, hairpin NAT scenarios, and the implications of NAT for microservice networking in cloud and container environments.
Exercise: Starter: Check the MTU of your network interfaces. On Linux:
ip link show. On Mac:ifconfig. On Windows:netsh interface ipv4 show interfaces. What is the MTU of your primary interface? Is it the standard 1,500 bytes?Intermediate: Use ping with a specific packet size to discover the path MTU to a destination. On Linux/Mac:
ping -s 1472 -M do google.com(1472 + 8 ICMP header + 20 IP header = 1,500). If it works, tryping -s 1473 -M do google.com- it should fail with a "message too long" error. Gradually reduce the size to find the exact path MTU. Try this across a VPN connection - what is the effective MTU?Advanced: Research PLPMTUD (Packetization Layer Path MTU Discovery, RFC 8899) - a newer approach that does not depend on ICMP and is therefore immune to PMTUD black holes. How does PLPMTUD probe the path MTU using transport-layer mechanisms instead of ICMP? How does QUIC (the protocol behind HTTP/3) implement its own PMTUD without relying on ICMP?