IPsec site-to-site: SMB copies hang after ~2 MB transferred — TCP window scaling vs firewall

Environment

Two-office IPsec tunnel (openswan/strongSwan on Linux 5.15 hosts), Windows 10 clients on remote LAN, SMB file shares hosted on a Samba server (Debian 11) inside Site A.

Symptoms

Large SMB file copies from Site B to Site A succeed for small files but stall partway: transfers stop progressing after ~2.0–2.4 MB and never complete; subsequent attempts either hang or drop the TCP connection after a long timeout.

Observations

ICMP (ping, traceroute) across the tunnel is stable with ~40 ms latency. Small TCP operations (SSH sessions, small HTTP downloads <1 MB) work fine. Wireshark capture on the Samba server shows a burst of packets then the remote TCP sender retransmits and stalls; the server's TCP window does not advance. TCP options show window scaling enabled. The IPsec tunnel MTU is 1400 on both ends. Disabling TCP window scaling on the Samba host (sysctl net.ipv4.tcp_window_scaling=0) temporarily allows the same large file transfer to complete, but this is not an acceptable permanent workaround.

Already attempted

Verified tunnel settings and re-established with rekey; increased IPsec MTU to 1420 on firewall (no change); toggled TCP autotuning on Samba host; swapped packet offload settings (ethtool -K) on the Linux host (GSO/TSO) — disabling GSO reduced the number of retransmits but did not fully fix; tried copying same file locally and over a non-VPN path successfully.

Question

What are the likely root causes for large TCP transfers stalling only across the IPsec tunnel, and what reversible diagnostic steps can identify whether this is PMTUD/MTU fragmentation, TCP window scaling interaction, or firewall/NAT path altering TCP options? What mitigation steps avoid disabling window scaling in production?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

TCP window scaling combined with large segments being handed to the IPsec layer and then fragmented in a way the remote stack/chipset mishandles. 85%

NIC offload (GSO/TSO/SG) producing large skbs that are not properly fragmented before encryption, exposing a kernel/driver bug. 70%

Network device (firewall/NAT) on the path altering or dropping TCP options or large fragmented packets. 65%

Summary

Symptoms point to a middlebox or encapsulation-induced fragmentation or a mismatch in TCP option handling across the encrypted path. The fact that disabling window scaling allows completion suggests an interaction between large segments and the tunnel's effective MTU or offload behavior. Disabling GSO/TSO changed retransmit characteristics, which implicates how outbound packets are segmented before encryption. Possible culprits include: (1) PMTUD blocked by firewalls dropping ICMP 'Fragmentation Needed' messages, (2) incorrect MTU accounting for IPsec overhead causing fragmentation of already-large TCP segments, (3) network equipment (NAT or firewall) rewriting or dropping TCP options, or (4) buggy NIC offloads producing oversized frames that the tunnel cannot handle.

Suggested Fixes

Diagnostic step 1

Capture full packet traces on both sides of the tunnel (pre- and post-IPsec endpoints) for a failing transfer to compare segment sizes, TCP options, and ICMP messages. Keep captures limited to necessary time windows to avoid PII exposure.

Diagnostic step 2

Temporarily set MTU at the Samba host and remote gateway incrementally lower (e.g., 1400 → 1350 → 1300) and retry transfers to check if reliable transfer size threshold changes; prefer reversible sysctl/ifconfig/ip link changes.

Diagnostic step 3

Enable IPsec DF handling/PMTU logging on both tunnel endpoints and check firewall logs for dropped ICMP Type 3 Code 4 messages; if ICMP is being blocked, allow ICMP 'Fragmentation Needed' across the path for diagnosis.