MongoDB primary stalls and throughput collapses when WiredTiger cache reaches high utilization during sustained bulk inserts

Environment

MongoDB 5.0 replica set (3 members) on Ubuntu 20.04 servers with 32GB RAM, WiredTiger cache configured to 60% of RAM (~18GB), NVMe-backed data directories; application performs sustained bulk inserts in batches of 10k documents.

Symptoms

Under sustained bulk insert load, the primary's insert throughput drops sharply after several minutes and clients observe increased write latency and occasional 'WriteConflict' errors. Server status shows WiredTiger cache utilization nearing the configured maximum; page eviction rates spike and fsync/write stall times increase. Secondary nodes fall behind replication as the primary struggles to keep up.

Observations

mongostat shows WT cache occupancy climbing to ~99% of the configured cache and eviction activity rising. mongod log contains repeated entries about 'WT cache exceeded target' and 'waiting for cache to be freed' followed by increased checkpoint times. iostat shows high write amplification and increased disk latency during stall periods. After pausing traffic the cache drains and throughput recovers, but the pattern repeats under resumed load.

Already attempted

Adjusted WiredTiger cache size up and down in small increments (restarted mongod to apply), which delayed but did not eliminate stalls. Reduced bulk insert batch size and observed smoother behavior but lower overall throughput. Enabled diagnostic log verbosity for WT and collected serverStatus metrics. Considered increasing RAM but constrained by server hardware. Implemented write-concern w:1 for bulk loads temporarily to reduce replication pressure (acceptable for test loads only).

Question

What are safe configuration changes and operational mitigations to prevent WiredTiger cache saturation and the resulting write stalls during large bulk insert bursts, and how should I instrument and tune the system to find a sustainable balance between throughput and eviction pressure?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

WiredTiger cache size too small relative to the working set produced by large bulk inserts, causing frequent eviction and checkpoints. 90%

Storage subsystem write latency/IOPS insufficient, so checkpoints cannot keep pace with dirty data generation. 85%

Bulk insert batching sizes are too large and create large spike working sets that transiently saturate cache. 80%

Summary

WiredTiger relies on an in-memory cache and background eviction to keep dirty pages within limits before checkpoints write to disk. Large, sustained bulk insert bursts can fill the cache faster than eviction/checkpoint can flush to disk, leading to stalls where insertion threads wait for cache pressure to relieve. The observed increase in checkpoint time and disk latency suggests the storage subsystem may be the limiting factor — either because write IOPS are insufficient or because insert batch size produces large working set churn. Short-term mitigations (smaller batches, lower write concern) reduce instantaneous cache pressure; longer-term fixes involve tuning WT cache/checkpoint settings, improving storage write performance, or staging bulk imports differently.

Suggested Fixes

Diagnostic step 1

Instrument: collect continuous serverStatus, WiredTiger metrics (cache bytes dirty, eviction server/threads, checkpoint times) and system-level metrics (iostat, vmstat) during a reproduce run to correlate cache growth with disk latency.

Diagnostic step 2

Reduce bulk insert batch size and stagger insert operations to smooth data ingestion rate; measure throughput vs latency tradeoff.

Diagnostic step 3

Consider increasing WiredTiger cache size if RAM is available, but only after measuring working set to avoid starving OS page cache. Apply changes in a maintenance window and monitor effects.