Redis replica stops applying updates after RDB mismatch during background save (BGSAVE)

Environment

Standalone Redis master (v6.2) and a chained replica on different physical host; master configured for AOF disabled, RDB snapshotting enabled (save 900 1). Replica connects as standard replica, persistent storage on both nodes is ext4 local disk.

Symptoms

Replica connects and syncs initially, then after the master performs a BGSAVE the replica reports replication lag growing and commands stop appearing on replica. Master continues to accept writes. Replica logs show 'Replication backlog error' followed by 'RDB file checksum mismatch' and replication enters partial-resync failure; replication only resumes after manually restarting the replica redis-server.

Observations

Master redis log contains 'Background saving terminated by signal 9' around the time of the issue. - Replica redis log shows 'Short read or RDB mismatch' and later 'Failed to load RDB'. - The master's dump.rdb exists but at times was observed sized 0 or truncated before BGSAVE termination. - Disk SMART and system logs show no clear disk I/O errors; however at one observed incident the master had automated periodic filesystem snapshot (LVM snapshot) triggered during BGSAVE. - Reproducing a BGSAVE while monitoring showed master spawn child write to dump.rdb; if an interruption occurs (OOM kill, signal), child terminates leaving truncated file.

Already attempted

Restarted replica to force full resync (worked temporarily). - Disabled scheduled LVM snapshots during a test window; triggered manual BGSAVE repeatedly — when no snapshot coincided, issue did not occur. - Increased ulimit for Redis process and monitored for OOM or SIGKILL — no clear OOM on master. - Moved dump.rdb to another disk on master (faster) temporarily and BGSAVE completed successfully during tests.

Question

What are the likely causes for the RDB write to be interrupted or corrupted on the master causing the replica to report an RDB checksum mismatch and stop applying updates? What safe, reversible mitigations can I apply to avoid replication stalls without losing data?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Concurrent filesystem operations (LVM snapshots, backup agents) interfering with the atomic rename pattern Redis uses for dump.rdb producing incomplete/truncated file 95%

BGSAVE child killed by OOM-killer or by administrator/system script during heavy memory/disk activity 60%

Disk or filesystem errors causing partially written dump.rdb (e.g., mounted with options that interact poorly with atomic replacements) 40%

Summary

A checksum mismatch on the RDB indicates the file written by the master's child process (spawned by BGSAVE) was truncated or corrupted before completion. Common causes include the BGSAVE child being terminated by signals (OOM-killer), interference by filesystem-level actions (snapshots, backup tools) operating on the same file, or disk I/O errors during the write. When the master creates a truncated dump.rdb, replicas attempting to do a full sync will detect the mismatch and fail to load the RDB, requiring manual intervention. The correlation with LVM snapshot activity suggests filesystem-level operations may be conflicting with Redis' child writes.

Suggested Fixes

Diagnostic step 1

Check system logs (kern.log, dmesg) on master for OOM killer or process signals at the time of BGSAVE termination (look for 'Out of memory' or SIGKILL)

Diagnostic step 2

Schedule LVM snapshots or backup jobs to avoid overlapping with Redis BGSAVE: either pause snapshots during backup window or configure snapshots to operate on a different file/volume; test BGSAVE repeatedly while snapshot disabled to confirm correlation

Diagnostic step 3

Enable Redis RDB and AOF coexistence temporarily (appendonly yes) as a mitigation so the replica can replay AOF if RDB load fails; note AOF has different performance characteristics—test in non-production first