1
Diagnostic step 1
Immediately stop the snapshot job that replaces /mnt/wal_store to prevent further interruptions; run on maintenance window.
FULL AI ANALYSIS
PostgreSQL WAL archiver fails with 'could not stat' after backup script replaced pg_wal with symlink
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
The immediate failure stems from the pg_wal directory being a symlink into a storage location that the backup snapshot job replaced or remounted, causing the symlink target to become invalid. PostgreSQL expects WAL segment files to exist at the path resolved by pg_wal; when they vanish mid-archive, archive_command fails and replication stalls. Potential root issues include backup job replacing mounts atomically, using LVM snapshot commands that temporarily unmount/mount targets, or recreating directories and changing inode references rather than operating on the live path. Restoring continuity requires re-establishing a stable pg_wal path with correct ownership/permissions and ensuring archive_command can atomically write files; careful recovery is necessary to avoid WAL gaps.
Possible causes
Backup/snapshot cron replaced or remounted the underlying filesystem where pg_wal symlink points, invalidating the symlink target at runtime.
100%pg_wal was moved to a different mount for capacity reasons but the mount is not persistent across the snapshot workflow, causing transient missing files.
80%archive_command expects direct file access but the underlying mount uses copy-on-write snapshots that create transient unavailability.
30%Permissions or SELinux/AppArmor prevented archive_command from creating files at the new target after snapshot remount, but logs show missing file rather than permission denied.
5%Suggested fixes
2
Diagnostic step 2
Avoid changing pg_wal path while PostgreSQL is running. If pg_wal must reside on alternate storage, perform an offline move following the documented procedure: stop PostgreSQL, move files to the new mount, update postgresql.conf (if using wal_directory symlink approach) or recreate pg_wal symlink, ensure ownership postgres:postgres and mode 700, then start PostgreSQL.
3
Diagnostic step 3
Temporarily re-create the missing WAL files by restoring from the most recent snapshot copy of /mnt/wal_store (do not fabricate WAL files). Place files at the symlink target path and ensure permissions match postgres:postgres.
4
Diagnostic step 4
Once WAL files are present, re-enable archive_command and monitor logs for successful archiving; use: tail -F /var/log/postgresql/postgresql-13-main.log and check for "archived WAL file" messages.
5
Diagnostic step 5
Modify the snapshot job: do not replace mounts or directories used live by PostgreSQL. Use LVM snapshots at the block level without unmounting, or snapshot the filesystem while leaving the live mount untouched. If atomic replacement is required, schedule snapshots when PostgreSQL is stopped or use pg_basebackup for consistent backups instead of manipulating live pg_wal storage.