Docker Swarm overlay network: intermittent inter-node packet loss when two overlay networks share the same subnet/VNI

Environment

Docker Swarm cluster (4 manager/worker mixed roles) running Docker Engine 20.10.8 on Ubuntu 18.04 hosts. Multiple overlay networks created for different stacks; some were created manually with --subnet and custom options. Under high inter-service traffic between containers on different hosts, flows intermittently stall and TCP resets observed between service tasks.

Symptoms

Services on the same node have consistent connectivity; services scheduled on different hosts show intermittent packet loss and retransmits. Application logs show timeouts for inter-service RPCs. 'docker service ps' shows no container OOMs or crashes. Reproduced reliably by generating sustained 1 Gbps-ish traffic between replicas.

Observations

tcpdump on host interfaces shows VXLAN (UDP 4789) packets but some VXLAN packets have unexpected VNI values. 'docker network ls' shows multiple overlay networks; two networks were created weeks apart with identical --subnet and overlapping CIDRs. 'docker network inspect' for both networks lists different Network IDs but overlapping IPAM pools. sysctl net.ipv4.conf.all.forwarding=1 is enabled. MTU on the overlay interfaces differs across hosts (some nodes have mtu 1450, others 1500) because of earlier manual changes.

Already attempted

Restarted docker daemon on a single node and re-tested (partial improvement). Removed one overlay network and recreated with a non-overlapping subnet on a single test node, but issue recurred when other nodes still had old network definitions. Rebalanced services to force placement changes (no resolution).

Question

Is subnet/VNI overlap across overlay networks likely to cause the observed inter-node packet loss in Docker Swarm, and what safe diagnostics and reversible remediation steps should I run to eliminate VNI or MTU mismatches before considering host/kernel changes?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

Two overlay networks were created with overlapping IPAM subnets causing ambiguous routing and VXLAN encapsulation collisions across peers 75%

MTU mismatch across hosts (1500 vs 1450) causing fragmentation or blackhole for encapsulated VXLAN packets, visible as intermittent stalls under high throughput 55%

Underlying switch or network equipment dropping or rate-limiting VXLAN/UDP encapsulated packets (multicast/unicast handling), especially when many overlay networks exist 30%

Summary

Overlay drivers use VXLAN with VNI assignments and IPAM subnets. Overlapping subnets across overlay networks can confuse local routing and VXLAN encapsulation, especially if multiple overlay networks exist with identical subnets or if the engine assigned conflicting VNIs. Inconsistent MTU values across nodes can cause fragmentation/blackhole for large frames. Observed unexpected VNI values in captures and duplicate IPAM pools strongly point toward subnet/VNI conflicts as the leading cause; MTU mismatch is a plausible secondary contributor to intermittent stalls under high throughput.

Suggested Fixes

Diagnostic step 1

Run 'docker network ls' and 'docker network inspect <network>' on every node; export outputs to compare IPAM pools and network IDs cluster-wide. Look for duplicated --subnet entries across different overlay networks.

Diagnostic step 2

On a quiet maintenance window, create a new test overlay network with a unique, non-overlapping subnet (e.g., 10.10.0.0/24) and deploy a pair of test containers on different nodes to verify cross-node connectivity. This is reversible (remove network after test).

Diagnostic step 3

Check MTU on each node's docker_gwbridge and veth/vxlan interfaces: 'ip link show' and 'ip addr' to confirm values. Normalize MTU across hosts (set to 1450 if underlying network requires headroom for encapsulation). Changing MTU is reversible but affects running traffic; schedule during low-traffic window.