PostgreSQL WAL archiver fails with 'could not stat' after backup script replaced pg_wal with symlink

Environment

PostgreSQL 13.5 running on Ubuntu 20.04 LTS; primary host: db-primary (4 vCPU, 8GB RAM); WAL archive_command used to copy files to /backups/wal; nightly filesystem snapshot job runs on secondary storage

Symptoms

WAL archiving stopped; primary emitted repeated archive errors and stopped recycling WAL segments; replication to standby stalled within 15 minutes of the issue appearing.

Observations

postgres log lines: "archive_command 'cp %p /backups/wal/%f' failed with exit code 1" and later "could not stat file "/var/lib/postgresql/13/main/pg_wal/0000000100000001000000A1": No such file or directory". ls -l /var/lib/postgresql/13/main/pg_wal shows pg_wal is a symlink to /mnt/wal_store/pg_wal; /mnt/wal_store is on an ext4 LVM snapshot target created by a cron job. The snapshot job replaced /mnt/wal_store with a new mount and recreated /mnt/wal_store/pg_wal as an empty directory, leaving the pg_wal symlink pointing to now-missing files. cp run manually on an existing WAL file fails with "No such file or directory". Owner and permissions on /backups/wal are postgres:postgres, 700.

Already attempted

Restarted PostgreSQL service (systemctl restart postgresql) — service started but archiving still failed. Verified standby still caught up only after manually copying missing WAL files from an older snapshot. Temporarily disabled archive_command in postgresql.conf and restarted to stop error spam. Inspected backup cron that manages snapshots but did not modify it yet.

Question

What is the safest way to restore WAL archiving quickly and prevent the snapshot job from breaking pg_wal again without compromising WAL integrity or losing replication continuity?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Backup/snapshot cron replaced or remounted the underlying filesystem where pg_wal symlink points, invalidating the symlink target at runtime. 100%

pg_wal was moved to a different mount for capacity reasons but the mount is not persistent across the snapshot workflow, causing transient missing files. 80%

archive_command expects direct file access but the underlying mount uses copy-on-write snapshots that create transient unavailability. 30%

Summary

The immediate failure stems from the pg_wal directory being a symlink into a storage location that the backup snapshot job replaced or remounted, causing the symlink target to become invalid. PostgreSQL expects WAL segment files to exist at the path resolved by pg_wal; when they vanish mid-archive, archive_command fails and replication stalls. Potential root issues include backup job replacing mounts atomically, using LVM snapshot commands that temporarily unmount/mount targets, or recreating directories and changing inode references rather than operating on the live path. Restoring continuity requires re-establishing a stable pg_wal path with correct ownership/permissions and ensuring archive_command can atomically write files; careful recovery is necessary to avoid WAL gaps.

Suggested Fixes

Diagnostic step 1

Immediately stop the snapshot job that replaces /mnt/wal_store to prevent further interruptions; run on maintenance window.

Diagnostic step 2

Avoid changing pg_wal path while PostgreSQL is running. If pg_wal must reside on alternate storage, perform an offline move following the documented procedure: stop PostgreSQL, move files to the new mount, update postgresql.conf (if using wal_directory symlink approach) or recreate pg_wal symlink, ensure ownership postgres:postgres and mode 700, then start PostgreSQL.

Diagnostic step 3

Temporarily re-create the missing WAL files by restoring from the most recent snapshot copy of /mnt/wal_store (do not fabricate WAL files). Place files at the symlink target path and ensure permissions match postgres:postgres.