1
Diagnostic step 1
Run 'docker network ls' and 'docker network inspect <network>' on every node; export outputs to compare IPAM pools and network IDs cluster-wide. Look for duplicated --subnet entries across different overlay networks.
FULL AI ANALYSIS
Docker Swarm overlay network: intermittent inter-node packet loss when two overlay networks share the same subnet/VNI
Generated Oct 14, 2026, 9:39:02 AM
Analysis summary
Overlay drivers use VXLAN with VNI assignments and IPAM subnets. Overlapping subnets across overlay networks can confuse local routing and VXLAN encapsulation, especially if multiple overlay networks exist with identical subnets or if the engine assigned conflicting VNIs. Inconsistent MTU values across nodes can cause fragmentation/blackhole for large frames. Observed unexpected VNI values in captures and duplicate IPAM pools strongly point toward subnet/VNI conflicts as the leading cause; MTU mismatch is a plausible secondary contributor to intermittent stalls under high throughput.
Possible causes
Two overlay networks were created with overlapping IPAM subnets causing ambiguous routing and VXLAN encapsulation collisions across peers
75%MTU mismatch across hosts (1500 vs 1450) causing fragmentation or blackhole for encapsulated VXLAN packets, visible as intermittent stalls under high throughput
55%Underlying switch or network equipment dropping or rate-limiting VXLAN/UDP encapsulated packets (multicast/unicast handling), especially when many overlay networks exist
30%Docker engine bug that reuses VNIs when networks are removed and re-created quickly across the cluster (less likely but possible depending on engine version)
20%Suggested fixes
2
Diagnostic step 2
On a quiet maintenance window, create a new test overlay network with a unique, non-overlapping subnet (e.g., 10.10.0.0/24) and deploy a pair of test containers on different nodes to verify cross-node connectivity. This is reversible (remove network after test).
3
Diagnostic step 3
Check MTU on each node's docker_gwbridge and veth/vxlan interfaces: 'ip link show' and 'ip addr' to confirm values. Normalize MTU across hosts (set to 1450 if underlying network requires headroom for encapsulation). Changing MTU is reversible but affects running traffic; schedule during low-traffic window.
4
Diagnostic step 4
Capture VXLAN/UDP traffic on nodes using tcpdump to confirm VNI and to correlate which VNI carries which overlay network's traffic: 'sudo tcpdump -i any -n -s 0 udp port 4789'. Avoid changing production networks while capturing.
5
Diagnostic step 5
If network equipment is suspected to drop VXLAN, coordinate with network team to verify switch ACLs, MTU, and whether UDP 4789 is allowed and not being re-written. If inspection indicates Docker engine VNI reuse or a possible engine bug, consider upgrading Docker to a stable later version in a staging environment and open a bug with Docker if issue persists.