PostgreSQL logical replication slot grows until disk full after subscriber disconnects during nightly load

Environment

PostgreSQL 13.8 on Debian 11 VM (8 vCPU, 32 GB RAM), primary and one logical replica on separate hosts; publisher uses pglogical extension for cross-datacenter replication; WAL archived to local filesystem; nightly bulk import job (COPY) runs 02:00–03:00.

Symptoms

Primary host runs out of disk space on /var/lib/postgresql/13/main/WAL disk within 24 hours of a subscriber network blip; application errors 'could not write to file' and new transactions fail with 'no space left on device'.

Observations

pg_replication_slots shows a slot with confirmed_flush_lsn stagnating several GB behind current WAL; pg_stat_wal_receiver on subscriber shows 'disconnected' during the incident window; archive_command continued to write WAL files; df -h shows WAL/pg_wal partition at 100%; backup logs show base backup completed prior to incident; no long-running transactions on primary according to pg_stat_activity snapshot gathered after space freed.

Already attempted

Stopped the subscriber service and manually removed old WAL files (mv to tmp on separate disk) to restore space; restarted primary Postgres to resume normal operations; verified pg_replication_slots output; attempted to drop unused logical slot with pg_drop_replication_slot but slot is active and drop failed until subscriber was cleanly stopped.

Question

What is causing the logical replication slot to retain WAL (growing until disk full) after the subscriber disconnects during the nightly load, and what safe steps can I take to recover without data loss and prevent recurrence?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

Subscriber disconnected or WAL receiver crashed during heavy WAL generation (nightly COPY) so replication slot on primary cannot advance and WAL accumulates. 85%

Subscriber is running out of resources (I/O or CPU) and is applying WAL too slowly during bulk load, causing confirmed_flush_lsn to lag behind and slots to retain WAL. 70%

pglogical logical decoding worker on subscriber hung due to schema drift or extension mismatch and stopped acknowledging WAL without clear error propagation. 55%

Summary

Symptoms match a replication slot that cannot advance because the subscriber is disconnected or stopped receiving WAL while the primary continues to produce large volumes of WAL (bulk COPY). Possible root causes include subscriber not consuming WAL due to network or resource constraints, WAL receiver crashes on subscriber retaining a restart_lsn, logical replication workers blocked by long transactions or conflicting schema changes, or misconfiguration where pglogical or logical decoding plugins stall. Manual deletion of WAL files frees space but is unsafe long-term. Recovery should prioritise preserving WAL needed for subscribers, confirm subscriber state, and implement measures (throttling, slot monitoring, retention limits) to prevent primary disk exhaustion.

Suggested Fixes

Diagnostic step 1

Do not rm or permanently delete WAL files that are still referenced by replication slots. If you must free space, move (mv) WAL files to a separate disk or compress them to a temporary location, but preserve them for potential replay.

Diagnostic step 2

On the subscriber host, check postgresql logs and system logs around the disconnect time for WAL receiver crashes, OOM, or I/O stalls; restart the subscriber Postgres cleanly and confirm it can connect and resume replication.

Diagnostic step 3

Run SELECT slot_name, plugin, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots; on primary and compare confirmed_flush_lsn to current WAL LSN to estimate backlog. On subscriber, check pg_stat_wal_receiver and pg_last_wal_receive_lsn() to confirm progress.