FULL AI ANALYSIS

MongoDB primary stalls and throughput collapses when WiredTiger cache reaches high utilization during sustained bulk inserts

Generated Oct 3, 2026, 12:35:03 AM

Analysis summary

WiredTiger relies on an in-memory cache and background eviction to keep dirty pages within limits before checkpoints write to disk. Large, sustained bulk insert bursts can fill the cache faster than eviction/checkpoint can flush to disk, leading to stalls where insertion threads wait for cache pressure to relieve. The observed increase in checkpoint time and disk latency suggests the storage subsystem may be the limiting factor — either because write IOPS are insufficient or because insert batch size produces large working set churn. Short-term mitigations (smaller batches, lower write concern) reduce instantaneous cache pressure; longer-term fixes involve tuning WT cache/checkpoint settings, improving storage write performance, or staging bulk imports differently.

Possible causes

WiredTiger cache size too small relative to the working set produced by large bulk inserts, causing frequent eviction and checkpoints.

90%

Storage subsystem write latency/IOPS insufficient, so checkpoints cannot keep pace with dirty data generation.

85%

Bulk insert batching sizes are too large and create large spike working sets that transiently saturate cache.

80%

Checkpoint or eviction configuration defaults not tuned for high sustained insert throughput on current hardware.

60%

Suggested fixes

1

Diagnostic step 1

Instrument: collect continuous serverStatus, WiredTiger metrics (cache bytes dirty, eviction server/threads, checkpoint times) and system-level metrics (iostat, vmstat) during a reproduce run to correlate cache growth with disk latency.

2

Diagnostic step 2

Reduce bulk insert batch size and stagger insert operations to smooth data ingestion rate; measure throughput vs latency tradeoff.

3

Diagnostic step 3

Consider increasing WiredTiger cache size if RAM is available, but only after measuring working set to avoid starving OS page cache. Apply changes in a maintenance window and monitor effects.

4

Diagnostic step 4

Tune checkpoint settings: increase 'checkpointDelaySecs' to allow larger intervals between checkpoints if storage can handle larger bursts, or reduce if long checkpoints are causing stalls — test carefully in staging.

5

Diagnostic step 5

If disk write throughput is the bottleneck, move data files to faster NVMe, use RAID/striping appropriate for write throughput, or provision faster disks. If hardware upgrade is not possible, implement a staged ingest pipeline (write to an intermediate queue or use smaller batches) to reduce instantaneous cache pressure.