1
Diagnostic step 1
Capture packets on both the inner interface (before vxlan encapsulation) and the physical NIC for failing flows to confirm differences in actual MTU and presence of DF bits; limit captures and redact payloads as needed.
FULL AI ANALYSIS
Docker Swarm overlay: intermittent packet loss between nodes after lowering MTU — vxlan fragmented frames dropped
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
Lowering MTU for encapsulated overlays can cause inner packets to be too large for the post-encapsulation MTU of the physical path. When the inner packet has DF set, routers will drop and send ICMP Fragmentation Needed — but if ICMP is blocked or offloads present create oversized skbs, packets can be dropped without informative feedback. Offloads like GSO/TFO can produce large skbs that bypass proper fragmentation before vxlan encap, and vxlan devices may not always handle segmentation correctly depending on kernel/NIC driver. The combination of lowered MTU, DF bit on inner packets, and NIC offload behavior explains the intermittent large-transfer failures.
Possible causes
Inner-packet DF flag combined with insufficient PMTU discovery across encapsulated path, causing routers to drop fragments and no ICMP feedback to the origin.
95%NIC/kernel offload features (GSO/TSO/GRO) producing oversized frames that bypass fragmentation before vxlan encapsulation and exceed physical MTU.
90%Asymmetric paths where one direction has lower MTU or devices drop fragmented vxlan packets, causing only some node pairs to fail.
60%Bug or misconfiguration in Docker's overlay implementation or kernel vxlan handling on specific kernel/NIC driver versions.
25%Suggested fixes
2
Diagnostic step 2
Temporarily raise the overlay MTU back to 1500 on a test network or set Docker's --mtu to a value that gives headroom for encapsulation (e.g., 1450 → 1500) if the upstream tunnel allows it; test impact.
3
Diagnostic step 3
Disable offloads (ethtool -K <iface> gso off tso off gro off) on problematic hosts as a reversible diagnostic; if this stabilizes traffic, schedule a controlled maintenance to test newer NIC drivers/firmware and kernel versions.
4
Diagnostic step 4
Ensure ICMP 'Fragmentation needed' messages are permitted between nodes and across the upstream tunnel for PMTUD to function; check firewall/ufw rules and cloud provider ACLs.
5
Diagnostic step 5
If behavior persists after offload and PMTUD checks, collect kernel version, iproute2, docker info, and NIC driver/firmware details and escalate to NIC vendor or open a Docker/kernel bug with minimal reproducer captures.