1
Diagnostic step 1
Do not immediately delete the registry in production. Back up the existing registry file first for forensic analysis: cp /var/lib/filebeat/registry /var/lib/filebeat/registry.bak.
FULL AI ANALYSIS
Filebeat stops shipping logs after registry file corrupts following abrupt host shutdown
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
Filebeat uses the registry to track read offsets; abrupt shutdown can leave registry in an inconsistent state (partial writes or duplicate entries). Removing the registry forces reprocessing, which recovers missing logs but can produce duplicates. Editing the registry manually is risky unless inode/offset relations are fully understood. Better approach is to restore a consistent registry from backup or use Filebeat's file_identifier and clean_* settings to control reprocessing, plus reliable shutdown procedures and filesystem settings to reduce risk.
Possible causes
Registry file (on same filesystem) was partially written or corrupted during the host's abrupt power loss, leaving inconsistent offsets and duplicate entries.
95%High write concurrency combined with buffered writes meant registry did not flush to disk before crash, losing last-known offsets.
80%Log rotation produced files with reused inode numbers and Filebeat's file identity matched incorrectly after restart, causing offset mismatches.
35%Filebeat version with known registry format bugs causing improper handling of registry during crash (less likely but possible).
20%Suggested fixes
2
Diagnostic step 2
Run Filebeat in debug mode (filebeat -e -d 'publish' and 'input') on a staging copy of the registry to observe how Filebeat interprets entries and which files it will harvest; perform this on a non-production copy to validate behavior.
3
Diagnostic step 3
If duplicates are acceptable short-term, remove or move the registry and restart Filebeat to recover missing logs, then use deduplication queries in Elasticsearch to clean duplicates later. If duplicates are unacceptable, restore registry from recent host backup if available.
4
Diagnostic step 4
To reduce risk going forward: (a) place registry on a filesystem with writeback guarantees or a small battery-backed storage, (b) enable short flush intervals (registry_flush: 1s) so Filebeat flushes its registry frequently, and (c) ensure graceful shutdown of Filebeat during host maintenance using systemd stop hooks.
5
Diagnostic step 5
If unsure how to reconcile registry entries safely or if this affects many hosts, escalate to the logging/SRE team so they can assist with registry repair, Elasticsearch deduplication strategies, and possible Filebeat upgrade that addresses any registry-format issues.