FULL AI ANALYSIS

IPsec site-to-site: SMB copies hang after ~2 MB transferred — TCP window scaling vs firewall

Generated Oct 3, 2026, 12:35:03 AM

Analysis summary

Symptoms point to a middlebox or encapsulation-induced fragmentation or a mismatch in TCP option handling across the encrypted path. The fact that disabling window scaling allows completion suggests an interaction between large segments and the tunnel's effective MTU or offload behavior. Disabling GSO/TSO changed retransmit characteristics, which implicates how outbound packets are segmented before encryption. Possible culprits include: (1) PMTUD blocked by firewalls dropping ICMP 'Fragmentation Needed' messages, (2) incorrect MTU accounting for IPsec overhead causing fragmentation of already-large TCP segments, (3) network equipment (NAT or firewall) rewriting or dropping TCP options, or (4) buggy NIC offloads producing oversized frames that the tunnel cannot handle.

Possible causes

TCP window scaling combined with large segments being handed to the IPsec layer and then fragmented in a way the remote stack/chipset mishandles.

85%

NIC offload (GSO/TSO/SG) producing large skbs that are not properly fragmented before encryption, exposing a kernel/driver bug.

70%

Network device (firewall/NAT) on the path altering or dropping TCP options or large fragmented packets.

65%

Path MTU underestimation or ICMP 'Fragmentation Needed' being blocked so endpoints send segments too large for the IPsec encapsulated path.

60%

Suggested fixes

Diagnostic step 1

Capture full packet traces on both sides of the tunnel (pre- and post-IPsec endpoints) for a failing transfer to compare segment sizes, TCP options, and ICMP messages. Keep captures limited to necessary time windows to avoid PII exposure.

Diagnostic step 2

Temporarily set MTU at the Samba host and remote gateway incrementally lower (e.g., 1400 → 1350 → 1300) and retry transfers to check if reliable transfer size threshold changes; prefer reversible sysctl/ifconfig/ip link changes.

Diagnostic step 3

Enable IPsec DF handling/PMTU logging on both tunnel endpoints and check firewall logs for dropped ICMP Type 3 Code 4 messages; if ICMP is being blocked, allow ICMP 'Fragmentation Needed' across the path for diagnosis.

Diagnostic step 4

Disable GSO/TSO/SG on the tunnel-facing NIC(s) (ethtool -K <iface> gso off tso off gro off) to see if behavior stabilizes; these changes are reversible and useful to isolate offload bugs.

Diagnostic step 5

Test with TCP window scaling disabled on one endpoint as a controlled diagnostic only; do not deploy this as a production fix. If this confirms interaction, collect kernel and ipsec stack versions and escalate to network vendor/kernel mailing list with captures if needed.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads