Login or Register
FULL AI ANALYSIS

WAL archiving stalls and primary reports 'could not flush file' during checkpoint under nightly bulk load

Generated Oct 3, 2026, 12:35:03 AM

Analysis summary

The immediate cause of the 'could not flush file' errors is inability to complete WAL write/flush operations within the database because the underlying storage or archiving path is slow or failing. Two separate storage layers are implicated: (1) the NFS archive target reporting 'No space left' prevented archive_command from moving WAL files, causing WAL accumulation; (2) SAN I/O writeback timeouts indicate the primary's local storage experienced high write latency, which during heavy checkpoints causes database fsync/flush operations to block and error. Both layers must be addressed: free, reliable archive target capacity and mitigate SAN latency under bulk writes (or throttle the PostgreSQL checkpoints/commit behavior).

Possible causes

Archive target (NFS) ran out of space or became unresponsive, causing archive_command to fail and WAL files to pile up, leading to long checkpoint operations.

90%

Shared SAN LUN experienced transient high write latency or writeback stalls under heavy sequential/random writes from the bulk import, causing local fsyncs to time out.

85%

archive_command or its wrapper doesn't handle transient NFS failures or space errors gracefully (no retry/backoff), allowing WAL accumulation to reach critical size.

60%

LVM thin provisioning on the VM host allowed the logical volume to appear available while underlying datastore reached capacity, causing unpredictable writeback behavior.

50%

Suggested fixes

1

Diagnostic step 1

Immediately ensure WAL archives have space: free space on NFS archive target, verify remote disk usage (df -h) and available inodes (df -i); do not delete WAL files on primary—only remove safely archived files from the archive server after verifying backups.

2

Diagnostic step 2

Switch archive_command to a robust wrapper that logs failures and retries with exponential backoff, and consider archiving to a local fast staging disk before pushing to NFS to decouple database I/O from network storage.

3

Diagnostic step 3

Temporarily increase max_wal_size and checkpoint_timeout to reduce checkpoint frequency during heavy imports; additionally set wal_compression = on to reduce WAL volume if CPU permits.

4

Diagnostic step 4

Run SAN and VM host-level storage diagnostics with the infrastructure team: check for LUN saturation, host-side thin provisioning overcommit, and initiator-side writeback settings. Capture iostat and vmstat across failure window and escalate to storage vendor/infra if latency persists.

5

Diagnostic step 5

If NFS flakiness continues, configure PostgreSQL to stream WAL to a standby (streaming replication) to keep local archiving decoupled, or use a redundant archive target. Escalate to system/storage administrators if underlying SAN or NFS issues are observed—do not continue high-load imports until storage stability is confirmed.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads