Site-to-site IPsec tunnel stalls large TCP transfers after disabling MSS clamp on edge firewall

Environment

OPNsense 22.1 firewall at HQ terminating IKEv2 IPsec tunnel to Cisco ASR at remote site; both ends use NAT on the WAN side; MTU on LAN interfaces set to 1500; encapsulation adds ~60 bytes of overhead; endpoints include Linux file servers (Ubuntu 20.04) and Windows file shares across the tunnel.

Symptoms

Large TCP transfers (scp, rsync, SMB multi-GB copies) start at expected throughput for ~20–40s, then stall with throughput dropping to near-zero while TCP session stays established. Small requests (HTTP, SSH commands) complete normally. Latency and ICMP pings remain stable during stalls. The problem appears only after disabling the firewall's TCP MSS clamping feature during configuration cleanup.

Observations

tcpdump on the local firewall shows outbound packets with DF set and sizes near 1400–1450 bytes. Remote capture shows no corresponding ACKs after the stall begins. Tracepath from client to server shows decreasing MTU and then stops reporting 'Fragmentation needed' ICMP messages. Firewall logs show ICMP type 3/code 4 (fragmentation needed) being dropped by a global ICMP filter. Enabling MSS clamp on the OPNsense box immediately restores stable large-transfer throughput in tests.

Already attempted

Re-enabled TCP MSS clamping on the edge firewall — transfers resumed normally. Manually reduced MTU on Linux server interfaces to 1400 and re-tested — transfers succeed. Temporarily allowed ICMP type 3 code 4 in firewall policy and observed that traceroute/path MTU discovery reports correct MTU and transfers complete. Rebooted both firewalls and verified same behavior. No changes were made to encryption algorithms or tunnel lifetimes.

Question

Why do large TCP transfers stall only when MSS clamping is disabled, and what is the safest long-term remediation that avoids lowering MTU on every host or weakening firewall filtering?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

Firewall globally blocks ICMP Type 3 Code 4 (Fragmentation needed), preventing PMTUD from working. 100%

MSS/TCP window clamping disabled on edge device; endpoints negotiate too-large MSS during TCP handshake. 95%

IPsec + NAT overhead reduces effective path MTU but endpoints still send full-size segments (DF set). 90%

Summary

This is a classic Path MTU Discovery (PMTUD) issue combined with IPsec/NAT encapsulation reducing the effective path MTU. Encapsulated packets exceed the actual path MTU; because DF is set, routers need to return ICMP 'fragmentation needed' messages to allow endpoints to reduce segment size. The edge/firewall was dropping those ICMP messages, so PMTUD failed and TCP flows transmitted packets that never got through (black hole). MSS clamping works around this by adjusting the TCP handshake MSS to a safe value, preventing large segments that would trigger fragmentation. NAT/encapsulation overhead and ICMP filtering are the main contributors. Hardware offload/acceleration bugs on some firewalls also sometimes mishandle DF packets inside IPsec tunnels and can mimic this behavior.

Suggested Fixes

Diagnostic step 1

Re-enable TCP MSS clamping on the IPsec endpoint(s) to mitigate immediately (reversible change).

Diagnostic step 2

Temporarily allow ICMP Type 3 Code 4 through the relevant firewall policies between the two sites and re-test PMTUD behavior; remove overly broad ICMP drops rather than allowing all ICMP.

Diagnostic step 3

Use tracepath or ping -M do -s to probe PMTU and verify endpoints can receive ICMP Fragmentation Needed messages end-to-end.