1
Diagnostic step 1
Capture packet traces simultaneously on both WAN interfaces and on the LAN-facing interface during a reproducer transfer (tcpdump -w) to see where SYN/ACK and subsequent segments travel; compare 5-tuple and NAT mappings.
FULL AI ANALYSIS
Dual‑WAN router sends large TCP transfers out the slower backup link despite primary health showing OK
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
Symptoms point to asymmetry between egress selection and return path handling for long TCP flows. The router's policy-based routing plus conntrack/NAT can cause established flows to be pinned to the interface that handled the handshake; if return packets arrive via a different ISP or the path reports MTU issues, the router or remote hosts can drop or stall TCP. The 'long‑flow' script and transient link flaps on ISP-A further suggest route failover logic may be aggressively migrating flows. Possible packet fragmentation (ICMP 'frag needed') hints at MTU/MSS issues that disproportionately impact large transfers.
Possible causes
Policy-based routing or custom long-flow script moves established connections to backup link after detecting perceived link flaps, creating asymmetric NAT and dropped packets.
90%Stateful firewall / conntrack entries pinned to the interface that handled the handshake; return traffic arriving via the other ISP is treated as invalid and dropped.
85%Path MTU black hole on ISP-A path causing large segments to be dropped; lack of ICMP forwarding prevents PMTUD, leading to retransmits and apparent failover.
80%ISP-A transient packet loss or micro-flaps causing health checks to briefly mark the link degraded and trigger failover logic for long flows only.
75%Suggested fixes
2
Diagnostic step 2
Temporarily disable the custom 'long-flow' policy/script and observe a single large transfer; if it stays on ISP-A and succeeds, the script is likely responsible. Re-enable after test.
3
Diagnostic step 3
Enable MSS clamping on the router for PPPoE/MPLS path (if applicable) or add an iptables SNAT rule for the test client to force egress via ISP-A and verify if transfer completes; revert SNAT after test.
4
Diagnostic step 4
Run a controlled iperf3 test over ISP-A with increasing window sizes and log retransmits/RTT; if ICMP 'Fragmentation Needed' appears, add MSS clamping or adjust MTU and re-test.
5
Diagnostic step 5
Lower health-check sensitivity (increase retry count / probe interval) temporarily to avoid aggressive failover during tests; document original settings so you can restore them exactly after verification. If evidence points to ISP path issues, escalate to ISP-A with packet captures and MTU test results.