MongoDB index build on primary causes secondaries to fall behind as oplog wraps

Environment

MongoDB 4.4 replica set (3 nodes) on Linux VMs with WiredTiger, collection with ~500M documents, primary handles write traffic during nightly batch imports while a createIndex is running on a large field.

Symptoms

During the index build on the primary, secondaries' replication lag grows from seconds to hours; rs.status() on secondaries shows replication lag increasing and apply delay. Secondary eventually reports RECOVERING or fails to catch up. Primary's CPU and disk IO are high; clients experience increased write latency.

Observations

rs.printReplicationInfo() reports oplog window smaller than duration of index build-related operations; the index build generates a large number of write operations that are replicated. Attempts to restart secondary do not help — it must perform a resync if it falls too far behind. The index build was initiated as a foreground operation in the operational window; build progress shows many resources consumed on primary. There is no recent backup taken prior to starting the index build.

Already attempted

Tried stepping down primary to let index build run on a different node — cluster remained unstable. Attempted to increase oplog size on the fly (temporary growth via file expansion not supported without recreation), attempted to throttle writes during index build (application-level change) with limited effect. Killed the index build to stop activity, but partial index state and writes left the cluster with inconsistent performance.

Question

How can we recover secondaries without full resync and safely build the index without causing replication failure? What reversible steps avoid data loss and long downtime?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Oplog size too small relative to volume of operations and index build duration. 95%

Index build type/flags cause a heavy stream of replicated writes (rather than an in-place build on secondaries). 90%

Index build performed on primary when it would have been safer to build on a secondary that can be resynced or stepped down. 80%

Summary

A large index build on the primary generated a substantial volume of replication operations that exceeded the duration of the configured oplog window on secondaries. When the oplog no longer contains the earliest needed operations, a secondary cannot catch up and requires resynchronization. In MongoDB 4.4, some index builds are replicated as operations that secondaries must apply; resource starvation (CPU, I/O) on secondaries worsens the situation. The root issues are insufficient oplog retention for the size/duration of the operation and resource saturation on secondaries.

Suggested Fixes

Diagnostic step 1

If index build is still running and cluster is degraded, stop accepting writes from the application if possible to let secondaries catch up. This is reversible and limits data churn.

Diagnostic step 2

Add temporary capacity: bring up an additional secondary (with a larger oplog) to act as a catch-up node or to run the index build there. This avoids impacting existing secondaries.

Diagnostic step 3

Consider building the index on a secondary by taking it offline (stop mongod), starting mongod in standalone mode against the data copy, create the index there, then reintroduce it to the set as a secondary so it will replicate less. This requires careful orchestration and backup.