1
Diagnostic step 1
Temporarily remove the arbiter (rs.reconfig to remove) and observe cluster stability; if elections stop, the arbiter is implicated—this is reversible.
FULL AI ANALYSIS
MongoDB replica set exhibits repeated elections every 7–12 minutes after adding a low-power arbiter
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
Frequent elections correlate with heartbeat failures and short TCP disconnects between the arbiter and voting nodes. Arbiters participate in the vote tally and their unstable connectivity can cause primaries to step down if they perceive insufficient majority. Observed packet loss and disconnects from the arbiter's AZ implicate network instability as the likely proximate cause rather than a MongoDB protocol bug. The arbiter being on a low-power instance in a different AZ increases susceptibility to transient network or resource issues.
Possible causes
Intermittent network packet loss or transient TCP disconnects between arbiter and voting nodes causing heartbeat failures and election triggers.
95%Underlying AWS infrastructure issues or VPC routing that intermittently disrupts connectivity for the arbiter's subnet.
80%Arbiter hosted in a lower-tier instance or under heavy host-level network virtualization load producing instability.
60%Replica set heartbeat configuration too aggressive for cross-AZ latency/loss characteristics (default timeouts too low).
30%Suggested fixes
2
Diagnostic step 2
Collect detailed mongod logs with increased verbosity for replication/heartbeat (setParameter logLevel) on all nodes and capture tcpdump on arbiter and a voting node during an election to confirm packet loss or RSTs.
3
Diagnostic step 3
If cross-AZ placement is required, increase heartbeatTimeoutMillis and electionTimeoutMillis in a controlled manner and test impact in a maintenance window—document changes for rollback.
4
Diagnostic step 4
Prefer placing arbiters in the same AZ as majority of voting nodes or avoid arbiters entirely by adding a lightweight voting secondary in the same AZ to preserve availability; arbiters should not be on low-performance instances in different AZs.
5
Diagnostic step 5
If network traces confirm AWS VPC or instance NIC issues, open a support case with AWS with pcap samples and CloudWatch metrics for NIC errors and instance host health to investigate underlying network instability.