Dual‑WAN router sends large TCP transfers out the slower backup link despite primary health showing OK

Environment

Small office EdgeRouter X (OpenWrt-derived OS) running policy-based routing and NAT on two ISPs: primary 500/50 Mbps (ISP-A) and secondary 100/20 Mbps (ISP-B). Router uses per-connection tracking with a custom script to move long-lived flows to the preferred interface.

Symptoms

Large TCP transfers (SCP, rsync, multi-GB HTTP downloads) consistently use the secondary ISP-B and perform at ~80 Mbps; small web requests and short-lived TCP connections use ISP-A and are fine. When forcing traffic out ISP-A manually, large transfers begin but stall or suffer extreme retransmits after a few MB.

Observations

Traceroute for long transfer shows egress via ISP-B; ip route and ip rule output show policy-based routing entries keyed by connection mark. conntrack entries for long-lived connections show NAT translations with original outif set to eth1 (ISP-B). tcpdump on both WAN interfaces shows SYNs and ACKs asymmetrically: ACKs for large flows returning via ISP-A are being dropped by stateful firewall on the router unless NAT mapping matches. MSS/MTU observed 1500 on both WANs; occasional ICMP Fragmentation Needed messages visible on ISP-A path but not forwarded to host. Router logs show transient 'link down' events for ISP-A at 1–2 minute intervals correlated with failover script logs.

Already attempted

Increased policy route preference metrics for ISP-A; disabled the custom long‑flow script; temporarily changed firewall conntrack timeout settings; toggled MSS clamping and disabled TCP offload on a test client; rebooted router and swapped WAN cables. Performed iperf3 point-to-point tests: small streams ok on ISP-A; large parallel streams seem to migrate to ISP-B unless using explicit SNAT rule. Contacted ISP-A support — they reported no carrier outage.

Question

What targeted diagnostics and reversible router configuration changes will confirm whether this is (a) asymmetric routing / connection tracking & NAT mismatch, (b) a path MTU / MSS black hole on ISP-A, or (c) a link‑quality flap triggering the long‑flow script? What safe tests can I run during working hours without disrupting users, and what router settings should I revert if they worsen connectivity?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Policy-based routing or custom long-flow script moves established connections to backup link after detecting perceived link flaps, creating asymmetric NAT and dropped packets. 90%

Stateful firewall / conntrack entries pinned to the interface that handled the handshake; return traffic arriving via the other ISP is treated as invalid and dropped. 85%

Path MTU black hole on ISP-A path causing large segments to be dropped; lack of ICMP forwarding prevents PMTUD, leading to retransmits and apparent failover. 80%

Summary

Symptoms point to asymmetry between egress selection and return path handling for long TCP flows. The router's policy-based routing plus conntrack/NAT can cause established flows to be pinned to the interface that handled the handshake; if return packets arrive via a different ISP or the path reports MTU issues, the router or remote hosts can drop or stall TCP. The 'long‑flow' script and transient link flaps on ISP-A further suggest route failover logic may be aggressively migrating flows. Possible packet fragmentation (ICMP 'frag needed') hints at MTU/MSS issues that disproportionately impact large transfers.

Suggested Fixes

Diagnostic step 1

Capture packet traces simultaneously on both WAN interfaces and on the LAN-facing interface during a reproducer transfer (tcpdump -w) to see where SYN/ACK and subsequent segments travel; compare 5-tuple and NAT mappings.

Diagnostic step 2

Temporarily disable the custom 'long-flow' policy/script and observe a single large transfer; if it stays on ISP-A and succeeds, the script is likely responsible. Re-enable after test.

Diagnostic step 3

Enable MSS clamping on the router for PPPoE/MPLS path (if applicable) or add an iptables SNAT rule for the test client to force egress via ISP-A and verify if transfer completes; revert SNAT after test.