Docker Swarm overlay: intermittent packet loss between nodes after lowering MTU — vxlan fragmented frames dropped

Environment

Docker Swarm cluster (Docker Engine 20.10) across three bare-metal hosts with Intel NICs, overlay driver using vxlan, recent change: global MTU lowered from 1500 to 1400 to accommodate a cloud provider tunnel. Hosts run Ubuntu 22.04, kube-like container workloads use overlay networks for north-south and east-west traffic.

Symptoms

Services on different nodes intermittently fail to reach each other; small RPCs succeed but larger HTTP POSTs or binary payloads intermittently time out or produce connection resets. Errors correlate with larger packet sizes; retries sometimes succeed. Packet loss appears only between particular node pairs.

Observations

docker network inspect shows overlay MTU set to 1400. Captures on host interfaces show large vxlan-encapsulated frames exceeding the physical interface MTU after being encapsulated, with DF bit set on inner packets. Linux dmesg on one host logs 'fragmentation needed and DF set' warnings. Disabling GRO on outer vxlan devices reduced error rate. Hosts where NIC offloads were left enabled exhibited more frequent drops. The issue reproduced after the MTU change; before lowering MTU, problems were not observed.

Already attempted

Rebooted Docker daemons and nodes; removed/recreated overlay networks; changed swarm mtu setting back to 1500 temporarily on test overlay (impractical cluster-wide due to upstream tunnel). Disabled GRO/GSO/TSO on vxlan interfaces and host NICs; observed improvement but not full resolution. Tested same workloads on a single host (no overlay) — no failures.

Question

What is the best way to diagnose and mitigate vxlan overlay fragmentation and DF-bit related packet drops in a Docker Swarm environment when MTU must be reduced for upstream tunnels? Which reversible network-level settings (offloads, MTU, fragmentation policy) should be adjusted first to restore service reliability, and when should NIC vendor escalation be considered?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Inner-packet DF flag combined with insufficient PMTU discovery across encapsulated path, causing routers to drop fragments and no ICMP feedback to the origin. 95%

NIC/kernel offload features (GSO/TSO/GRO) producing oversized frames that bypass fragmentation before vxlan encapsulation and exceed physical MTU. 90%

Asymmetric paths where one direction has lower MTU or devices drop fragmented vxlan packets, causing only some node pairs to fail. 60%

Summary

Lowering MTU for encapsulated overlays can cause inner packets to be too large for the post-encapsulation MTU of the physical path. When the inner packet has DF set, routers will drop and send ICMP Fragmentation Needed — but if ICMP is blocked or offloads present create oversized skbs, packets can be dropped without informative feedback. Offloads like GSO/TFO can produce large skbs that bypass proper fragmentation before vxlan encap, and vxlan devices may not always handle segmentation correctly depending on kernel/NIC driver. The combination of lowered MTU, DF bit on inner packets, and NIC offload behavior explains the intermittent large-transfer failures.

Suggested Fixes

Diagnostic step 1

Capture packets on both the inner interface (before vxlan encapsulation) and the physical NIC for failing flows to confirm differences in actual MTU and presence of DF bits; limit captures and redact payloads as needed.

Diagnostic step 2

Temporarily raise the overlay MTU back to 1500 on a test network or set Docker's --mtu to a value that gives headroom for encapsulation (e.g., 1450 → 1500) if the upstream tunnel allows it; test impact.

Diagnostic step 3

Disable offloads (ethtool -K <iface> gso off tso off gro off) on problematic hosts as a reversible diagnostic; if this stabilizes traffic, schedule a controlled maintenance to test newer NIC drivers/firmware and kernel versions.