Filebeat stops shipping logs after registry file corrupts following abrupt host shutdown

Environment

Filebeat 7.17 installed on multiple Ubuntu 20.04 servers; shipping logs to Elasticsearch 7.17. Filebeat registry located at /var/lib/filebeat/registry. One host experienced abrupt power loss during heavy log write.

Symptoms

After the unclean shutdown, Filebeat reports harvesting files but Elasticsearch shows no new documents. Rebooted host and Filebeat starts but some log files are not being processed (older log rotations appear to be skipped). No obvious Filebeat crashes; service stays up.

Observations

Filebeat logs (journalctl -u filebeat) show 'Start harvester for new file' entries but subsequent 'file already finished' or 'current offset greater than file size' warnings for some rotated files. Inspecting /var/lib/filebeat/registry reveals entries referencing offsets larger than the file sizes, and duplicate entries for the same inode/path with different offsets. Removing the registry file and restarting Filebeat causes re-harvest of older files but risks duplicates in Elasticsearch and loss of exactly-once semantics. File permissions are intact.

Already attempted

Restarted Filebeat service (no change). Backed up and removed registry file to force full re-harvest — this successfully shipped the missing logs but created duplicates for recent entries in Elasticsearch. Tried setting clean_inactive and close_removed settings lower, and ran filebeat -e -d '*' to get debug output. Considered manually editing registry but worried about corrupting offsets and causing duplicates or gaps.

Question

What is the safest recovery path to restore continuous log shipping without creating duplicates or missing events? How can the registry be repaired or reconciled, and what preventive configuration changes reduce the risk of registry corruption after abrupt shutdowns?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Registry file (on same filesystem) was partially written or corrupted during the host's abrupt power loss, leaving inconsistent offsets and duplicate entries. 95%

High write concurrency combined with buffered writes meant registry did not flush to disk before crash, losing last-known offsets. 80%

Log rotation produced files with reused inode numbers and Filebeat's file identity matched incorrectly after restart, causing offset mismatches. 35%

Summary

Filebeat uses the registry to track read offsets; abrupt shutdown can leave registry in an inconsistent state (partial writes or duplicate entries). Removing the registry forces reprocessing, which recovers missing logs but can produce duplicates. Editing the registry manually is risky unless inode/offset relations are fully understood. Better approach is to restore a consistent registry from backup or use Filebeat's file_identifier and clean_* settings to control reprocessing, plus reliable shutdown procedures and filesystem settings to reduce risk.

Suggested Fixes

Diagnostic step 1

Do not immediately delete the registry in production. Back up the existing registry file first for forensic analysis: cp /var/lib/filebeat/registry /var/lib/filebeat/registry.bak.

Diagnostic step 2

Run Filebeat in debug mode (filebeat -e -d 'publish' and 'input') on a staging copy of the registry to observe how Filebeat interprets entries and which files it will harvest; perform this on a non-production copy to validate behavior.

Diagnostic step 3

If duplicates are acceptable short-term, remove or move the registry and restart Filebeat to recover missing logs, then use deduplication queries in Elasticsearch to clean duplicates later. If duplicates are unacceptable, restore registry from recent host backup if available.