Login or Register
FULL AI ANALYSIS

MongoDB secondary stops applying oplog entries after intermittent TLS handshake errors between shard nodes

Generated Oct 3, 2026, 12:35:03 AM

Analysis summary

Evidence suggests two overlapping issues: cert rotation introduced a sensitive TLS verification path (rollback fixed one set), and network instability (cross-AZ latency/packet loss) causes TLS handshakes to fail intermittently. MongoDB replication relies on persistent connections; repeated handshake failures can cause secondaries to stop applying oplog data. A robust approach combines validating cert chain/trust across all nodes and hardening network/TLS settings to tolerate transient failures.

Possible causes

Misconfigured or partially-applied certificate rotation produced trust mismatches between some nodes leading to TLS verification failures.

90%

Transient cross-AZ network instability (latency spikes, packet loss) causing TLS handshakes to fail or time out.

75%

MongoDB/TLS library sensitivity to handshake timeouts; insufficient retry/backoff allows replication to stop.

60%

MTU/path-MTU issues causing TLS records to be dropped or connections reset during handshake.

35%

Suggested fixes

1

Diagnostic step 1

Collect synchronized logs and packet captures across affected replica set members during an event to correlate TLS handshake failures with network errors (use tcpdump with --immediate/PCAP timestamps; keep captures limited to avoid data leak).

2

Diagnostic step 2

Verify certificate chain on all nodes with openssl s_client and confirm CRL/OCSP settings and CA trust; if rotation was partial, perform coordinated, documented redeploy of certs across the entire cluster (do this during maintenance window).

3

Diagnostic step 3

Temporarily increase MongoDB's net.tlsHandshakeTimeout and connection retry settings and tune OS TCP keepalive so replicas can survive short network blips—make these changes in staging first.

4

Diagnostic step 4

Check and align MTU across hosts and cloud networking devices; enable MSS clamping on intermediate routers if fragmentation is suspected.

5

Diagnostic step 5

If instability persists, consider colocating critical replica members in the same AZ or increasing voting/replica set topology to reduce cross-AZ replication dependency until network is stable; escalate to cloud network support for persistent cross-AZ packet loss diagnostics.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads