Postfix STARTTLS deliveries intermittently fail with TLS handshake timeouts and RST from upstream gateway

Environment

Small office: Ubuntu 20.04 Postfix mail server (192.168.10.10) behind an ISP‑supplied modem in passthrough to a pfSense 2.5 router; outbound SMTP uses port 587 with STARTTLS to multiple external MX hosts.

Symptoms

Outgoing mail to some destinations queues and retries with 'TLS handshake timeout' or 'lost connection after STARTTLS'; other destinations deliver successfully. Deliveries that fail from the office succeed when retried from a remote VPS.

Observations

tcpdump on the mailserver shows TCP SYN,SYN-ACK, ClientHello sent but no ServerHello; occasionally the remote IP responds with RST after several retransmits. Traceroute and mtr to affected MX hosts show no routing changes. Path MTU probe (tracepath) indicates an MTU mismatch on the path for affected destinations. Postfix logs show consistent STARTTLS failure messages for the same subset of receiving hosts.

Already attempted

Disabled SIP/ALG on the modem, enabled MSS clamping on pfSense, toggled firewall SPI, tested direct connection (bypassing pfSense) which reduced but did not eliminate failures, updated pfSense and modem firmware, retested with openssl s_client and experienced the same handshake timeouts for failing hosts.

Question

What are the likely causes of intermittent STARTTLS/TCP handshake failures only for some MX hosts from behind a consumer gateway + router, and which reversible diagnostics and mitigations should I run before involving the ISP?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

PMTU black hole: intermediate device drops ICMP 'fragmentation needed' or fragments TLS handshake packets. 90%

No MSS clamping on NAT device causing packets larger than path MTU to be sent and dropped. 75%

ISP modem or middlebox implementing an ALG/NAT that mishandles STARTTLS/TLS fragmentation. 60%

Summary

Symptoms and packet traces point to a TCP level problem (packets being dropped or fragmented) rather than Postfix TLS configuration. The fact that only some MX hosts are affected and that tracepath highlights MTU inconsistency suggests a PMTU black‑hole or incorrect MSS handling on the modem/router path. Consumer gateways (or ISP middleboxes) sometimes mishandle fragmented TCP packets or perform transparent proxies/ALGs that interfere with long TLS handshakes. The successful delivery from an external VPS indicates the remote MX hosts are fine.

Suggested Fixes

Diagnostic step 1

Collect a full pcap of a failing TCP session (capture on the Postfix host and on the router if possible) around the ClientHello to document missing ServerHello or RST packets.

Diagnostic step 2

Confirm Path MTU to the failing MX using tracepath and note where 'pmtu' changes occur; try ping -M do -s <size> to identify max working packet size (reversible).

Diagnostic step 3

Enable MSS clamping on the pfSense WAN rule (set MSS to 1360) to work around PMTU black holes and retest deliveries.