1
Diagnostic step 1
Collect a full pcap of a failing TCP session (capture on the Postfix host and on the router if possible) around the ClientHello to document missing ServerHello or RST packets.
FULL AI ANALYSIS
Postfix STARTTLS deliveries intermittently fail with TLS handshake timeouts and RST from upstream gateway
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
Symptoms and packet traces point to a TCP level problem (packets being dropped or fragmented) rather than Postfix TLS configuration. The fact that only some MX hosts are affected and that tracepath highlights MTU inconsistency suggests a PMTU black‑hole or incorrect MSS handling on the modem/router path. Consumer gateways (or ISP middleboxes) sometimes mishandle fragmented TCP packets or perform transparent proxies/ALGs that interfere with long TLS handshakes. The successful delivery from an external VPS indicates the remote MX hosts are fine.
Possible causes
PMTU black hole: intermediate device drops ICMP 'fragmentation needed' or fragments TLS handshake packets.
90%No MSS clamping on NAT device causing packets larger than path MTU to be sent and dropped.
75%ISP modem or middlebox implementing an ALG/NAT that mishandles STARTTLS/TLS fragmentation.
60%Intermittent stateful firewall on ISP path closing long TLS handshakes or large initial windows.
35%Suggested fixes
2
Diagnostic step 2
Confirm Path MTU to the failing MX using tracepath and note where 'pmtu' changes occur; try ping -M do -s <size> to identify max working packet size (reversible).
3
Diagnostic step 3
Enable MSS clamping on the pfSense WAN rule (set MSS to 1360) to work around PMTU black holes and retest deliveries.
4
Diagnostic step 4
Test SMTP over TLS from the mailserver to the destination using openssl s_client -starttls smtp -connect host:25 (or :587) and capture the handshake to reproduce the failure.
5
Diagnostic step 5
Temporarily force Postfix to use smaller TCP segments (SO_SNDBUF/SO_RCVBUF tuning) or lower TLS record size at client side (if supported) to avoid fragmentation while testing; restore defaults afterwards if not helpful. If the pcap shows the modem or ISP device sending RSTs or dropping ICMP fragments after MSS clamping fails, escalate to the ISP with the pcap and tracepath output for them to investigate; provide timestamps and packet captures rather than raw logs.