MongoDB secondary stops applying oplog entries after intermittent TLS handshake errors between shard nodes

Environment

Sharded MongoDB 6.0 cluster: three replica sets for shards, each with primary and two secondaries. Internal cluster traffic uses TLS mutual authentication (certs rotated from an internal CA). Observed on AWS VPC across three AZs with VPC peering to a backup site.

Symptoms

A secondary node reports replication lag that grows and replication stops applying new operations; rs.status shows SECONDARY but recent oplog entries are not applied. The mongod log contains frequent TLS handshake failures (message: 'SSL peer certificate verification failed' and 'error during TLS handshake'). Manual rs.syncFrom to another member temporarily repairs state but lag returns.

Observations

TLS certificates on all members were rotated two days before issue appeared; timestamps and CA chain appear correct when inspected. Network-level packet captures show TLS ClientHello and ServerHello followed by connection resets from time to time between particular AZs. The issue is intermittent and correlates with higher cross-AZ latency spikes reported by cloud provider metrics. When the secondary is promoted to primary in tests, it functions normally until it later becomes secondary again and the problem recurs.

Already attempted

Rolled back to previous certs on one replica set — replication stabilized there. Increased the TLS handshake timeout settings in mongod config — provided limited improvement. Adjusted keepalive values at OS level — no durable change. Replaced network NIC on the affected host — problem persisted on other hosts in same AZ during latency spikes.

Question

How to reliably diagnose whether TLS handshake resets stem from certificate validation issues, network MTU/packet loss, or ephemeral cloud network path instability, and what safe mitigations (config or topology) can keep secondaries applying oplog entries during intermittent TLS failures?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Misconfigured or partially-applied certificate rotation produced trust mismatches between some nodes leading to TLS verification failures. 90%

Transient cross-AZ network instability (latency spikes, packet loss) causing TLS handshakes to fail or time out. 75%

MongoDB/TLS library sensitivity to handshake timeouts; insufficient retry/backoff allows replication to stop. 60%

Summary

Evidence suggests two overlapping issues: cert rotation introduced a sensitive TLS verification path (rollback fixed one set), and network instability (cross-AZ latency/packet loss) causes TLS handshakes to fail intermittently. MongoDB replication relies on persistent connections; repeated handshake failures can cause secondaries to stop applying oplog data. A robust approach combines validating cert chain/trust across all nodes and hardening network/TLS settings to tolerate transient failures.

Suggested Fixes

Diagnostic step 1

Collect synchronized logs and packet captures across affected replica set members during an event to correlate TLS handshake failures with network errors (use tcpdump with --immediate/PCAP timestamps; keep captures limited to avoid data leak).

Diagnostic step 2

Verify certificate chain on all nodes with openssl s_client and confirm CRL/OCSP settings and CA trust; if rotation was partial, perform coordinated, documented redeploy of certs across the entire cluster (do this during maintenance window).

Diagnostic step 3

Temporarily increase MongoDB's net.tlsHandshakeTimeout and connection retry settings and tune OS TCP keepalive so replicas can survive short network blips—make these changes in staging first.