FULL AI ANALYSIS

PostgreSQL logical replication slot grows until disk full after subscriber disconnects during nightly load

Generated Oct 14, 2026, 9:39:02 AM

Analysis summary

Symptoms match a replication slot that cannot advance because the subscriber is disconnected or stopped receiving WAL while the primary continues to produce large volumes of WAL (bulk COPY). Possible root causes include subscriber not consuming WAL due to network or resource constraints, WAL receiver crashes on subscriber retaining a restart_lsn, logical replication workers blocked by long transactions or conflicting schema changes, or misconfiguration where pglogical or logical decoding plugins stall. Manual deletion of WAL files frees space but is unsafe long-term. Recovery should prioritise preserving WAL needed for subscribers, confirm subscriber state, and implement measures (throttling, slot monitoring, retention limits) to prevent primary disk exhaustion.

Possible causes

Subscriber disconnected or WAL receiver crashed during heavy WAL generation (nightly COPY) so replication slot on primary cannot advance and WAL accumulates.

85%

Subscriber is running out of resources (I/O or CPU) and is applying WAL too slowly during bulk load, causing confirmed_flush_lsn to lag behind and slots to retain WAL.

70%

pglogical logical decoding worker on subscriber hung due to schema drift or extension mismatch and stopped acknowledging WAL without clear error propagation.

55%

Archive/retention misconfiguration: WAL files being archived but replication slot still requires them; local cleanup removed active WALs earlier causing transient failures and confusing state.

30%

Suggested fixes

1

Diagnostic step 1

Do not rm or permanently delete WAL files that are still referenced by replication slots. If you must free space, move (mv) WAL files to a separate disk or compress them to a temporary location, but preserve them for potential replay.

2

Diagnostic step 2

On the subscriber host, check postgresql logs and system logs around the disconnect time for WAL receiver crashes, OOM, or I/O stalls; restart the subscriber Postgres cleanly and confirm it can connect and resume replication.

3

Diagnostic step 3

Run SELECT slot_name, plugin, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots; on primary and compare confirmed_flush_lsn to current WAL LSN to estimate backlog. On subscriber, check pg_stat_wal_receiver and pg_last_wal_receive_lsn() to confirm progress.

4

Diagnostic step 4

If subscriber is permanently lost and you have a recent base backup, consider creating a new slot and/or reinitialising the subscriber via pg_basebackup; when reinitialising, drop the old slot only after ensuring subscriber re-creation or that no other consumers need it.

5

Diagnostic step 5

Implement slot monitoring and alerting (e.g., warn if WAL retention > X GB or confirmed_flush_lsn behind > Y MB). Consider setting max_slot_wal_keep_size (Postgres 13+) to limit WAL retention impact, or schedule base backups more frequently during heavy loads to allow safe WAL removal through archiving policies (test first). Read-only delay: throttle bulk loads or run them on replicas to avoid overwhelming replication consumers during peak times where possible (reconfigure COPY batching). Escalate to DBA or vendor support if logical decoding/pglogical shows internal errors or if WAL records required by slot were accidentally removed.