HTTP/HTTPS downloads stall ~65s over site-to-site GRE tunnel — TCP handshake OK then throughput drops to zero

Environment

Two-site setup: Site A uses Debian 11 iproute2 router (GRE tunnel endpoint 10.0.0.1) and Site B uses VyOS-based router (GRE endpoint 10.0.0.2). Underlay is ISP-managed routed WAN with MTU 1500. GRE tunnel configured with mtu 1476 on both ends to account for GRE+IP overhead. Downstream clients are Windows 10 and Ubuntu 22.04 desktops behind a NATed LAN at Site B. Large HTTP/HTTPS downloads originate from a public web server on the Internet and traverse the GRE tunnel between sites.

Symptoms

Large file downloads (HTTP/HTTPS >10 MB) from Internet servers start normally, TCP three‑way handshake completes and initial segments transfer. After roughly 60–75 seconds of transfer the receive throughput abruptly drops to zero; TCP retransmits begin and the session either times out after several minutes or resets. Small transfers (<1 MB) and browsing are unaffected. The stall is reproducible from multiple client machines behind Site B router and from files hosted on different external CDNs.

Observations

tcpdump on Site B GRE endpoint shows normal SYN/SYN-ACK/ACK and a stream of pushed segments, then long gaps with repeated ACKs and occasional retransmits from the client. Server-to-client packets appear to stop arriving at the GRE endpoint while client continues to ACK what it has received. ping -M do -s with sizes above 1400 bytes from Site B to a test server fails (no ICMP fragmentation-needed returned). tracepath from Site B to Internet hosts reports decreasing MTU after leaving the WAN but no ICMP fragmentation-needed messages are observed hitting the GRE endpoints. iptables NAT/forward rules on the Debian router do not explicitly modify MSS. Disabling the GRE tunnel and routing clients directly (temporary test) removes the stall — large transfers complete normally. Enabling kernel tcp_mtu_probing=1 on Site B reduces frequency of stalls but does not eliminate them.

Already attempted

Captured packet traces at both GRE endpoints and client; compared sequence numbers and timestamps. Temporarily lowed MTU on GRE interface to 1400 on both routers — transfers succeed for a test file. Turned on iptables TCPMSS CLAMP to PMTU on the Debian router; large downloads succeed while clamp is active. Disabled TCP offloads (TSO/GSO) on client NICs — no change when tunnel MTU unchanged. Replaced GRE with IP-in-IP tunnel for one test; same stall observed. Contacted ISP support — they report no uplink errors but would not confirm blocking of ICMP type 3. Restoring GRE MTU to 1476 without MSS clamp reproduces the stall.

Question

Given the packet captures and that lowering tunnel MTU or enabling MSS clamping prevents the stall, what are the likely root causes (routing/middlebox/MTU/ICMP handling) and which safe, reversible configuration changes should be applied on the Debian and VyOS routers to mitigate this without disrupting production? When is escalation to the ISP or deeper kernel-level debugging indicated?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

Intermediate WAN/ISP device is dropping packets larger than a given MTU and suppressing/silently dropping ICMP 'fragmentation needed' responses (PMTU black hole). 95%

WAN firewall or NAT device erroneously strips or blocks ICMP type 3 code 4 making PMTU discovery fail. 90%

GRE tunnel MTU set too high relative to actual path MTU (off-by-overhead or additional encapsulation upstream). 85%

Summary

Behavior matches a PMTU/MSS black‑hole: TCP connections establish and transfer until a packet larger than an intermediate MTU must be sent with DF set; if an intermediate hop drops those large packets and does not send ICMP fragmentation‑needed back to the sender (or those ICMPs are dropped on path), the sender never learns to reduce MSS and the flow stalls. The fact that lowering tunnel MTU or enabling TCP MSS clamping (which caps MSS to effective PMTU) fixes the issue points to either: (a) a mismatched MTU/overhead calculation on the GRE endpoints vs intermediate hop limits, (b) an ISP or WAN middlebox dropping ICMP type 3 code 4, or (c) NAT/conntrack hardware that mishandles DF/fragmentation. TCP offloads can exacerbate timing but are less likely the primary cause here.

Suggested Fixes

Diagnostic step 1

Reproduce with controlled tests: run tracepath (or ping -M do -s) from both GRE endpoints and capture on both endpoints for a failing transfer to confirm missing ICMP fragmentation-needed messages. Keep capture window focused to avoid PII.

Diagnostic step 2

Apply reversible mitigation: enable TCP MSS clamping on the GRE egress at both routers (iptables/ebtables rule: --clamp-mss-to-pmtu) and verify large transfers succeed; this avoids touching client stacks and is quickly reversible.

Diagnostic step 3

Temporarily lower the GRE interface MTU (e.g., to 1400) on both endpoints during business hours to confirm behavior change; document and revert if not acceptable for performance.