MongoDB replica set exhibits repeated elections every 7–12 minutes after adding a low-power arbiter

Environment

Three-node MongoDB replica set: primary (m5.large, AWS us-east-1), secondary (m5.large), and a third node added as an arbiter on a low-power t3.small instance in a different AZ. MongoDB 4.4.10, WiredTiger, journaling enabled. Network between AZs over AWS VPC peering.

Symptoms

After adding the arbiter, the replica set performs a primary election roughly every 7–12 minutes. During elections, writes fail with 'not master' until a new primary is elected (typically the previous primary rejoins after a short interval). Application reports transient write errors and occasional increased latency during failover windows.

Observations

mongod logs show 'no healthy members in set', 'election triggered', and 'heartbeat failed' messages with timestamps clustered around election events. The arbiter logs show frequent short TCP disconnects to primaries around the same times; network monitoring shows packet retransmits between the arbiter and the primary on the arbiter's AZ network interface. When the arbiter is stopped, elections cease and the cluster is stable. Ping and traceroute between nodes show sub-10ms latency but intermittent packet loss (0.5–1%) observed from arbiter to other nodes using continuous ping over 24h.

Already attempted

Removed any firewall rules between nodes; restarted mongod on arbiter and primaries; increased replication heartbeatTimeoutMillis in replica set configuration from default to 10000 ms (no improvement); monitored CloudWatch network metrics (no significant CPU spikes on arbiter). Temporarily relocated arbiter to same AZ as primary and observed improved stability for one day before similar election churn returned. Replaced t3.small with t3.medium for higher baseline CPU — did not resolve elections.

Question

What targeted diagnostics and configuration changes can confirm whether the arbiter's intermittent connectivity is causing election churn or whether there is a replica set configuration/timing parameter that needs tuning? Is using an arbiter across AZs acceptable, and what are recommended practices for arbiter placement to avoid split-brain behaviours?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Intermittent network packet loss or transient TCP disconnects between arbiter and voting nodes causing heartbeat failures and election triggers. 95%

Underlying AWS infrastructure issues or VPC routing that intermittently disrupts connectivity for the arbiter's subnet. 80%

Arbiter hosted in a lower-tier instance or under heavy host-level network virtualization load producing instability. 60%

Summary

Frequent elections correlate with heartbeat failures and short TCP disconnects between the arbiter and voting nodes. Arbiters participate in the vote tally and their unstable connectivity can cause primaries to step down if they perceive insufficient majority. Observed packet loss and disconnects from the arbiter's AZ implicate network instability as the likely proximate cause rather than a MongoDB protocol bug. The arbiter being on a low-power instance in a different AZ increases susceptibility to transient network or resource issues.

Suggested Fixes

Diagnostic step 1

Temporarily remove the arbiter (rs.reconfig to remove) and observe cluster stability; if elections stop, the arbiter is implicated—this is reversible.

Diagnostic step 2

Collect detailed mongod logs with increased verbosity for replication/heartbeat (setParameter logLevel) on all nodes and capture tcpdump on arbiter and a voting node during an election to confirm packet loss or RSTs.

Diagnostic step 3

If cross-AZ placement is required, increase heartbeatTimeoutMillis and electionTimeoutMillis in a controlled manner and test impact in a maintenance window—document changes for rollback.