1
Diagnostic step 1
Do not run further automated reallocation that could overwrite remaining copies. Identify which node currently contains the most up-to-date copy of each affected shard using 'CAT SHARDS' and node logs.
FULL AI ANALYSIS
Elasticsearch shard stuck INITIALIZING after disk-pressure eviction; recovery keeps failing with checksum mismatch
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
Repeated checksum mismatch and corrupt file messages during recovery indicate underlying segment files on the source or target node are damaged or inconsistent. Disk pressure likely caused partial writes or filesystem-level corruption. Options are limited: if corruption affects primary shard files, recovery from replicas or snapshot is required; if replicas are also corrupted, snapshot restore is safest. Using 'allocate_empty_primary' forces recovery but loses the shard's data; salvage tools are risky. It's necessary to determine whether any healthy copies exist on other nodes or in snapshot.
Possible causes
Hardware issues (failing SSD/HDD) producing IO errors and corrupted segments.
90%Filesystem-level corruption on the node that hosted the primary shard caused segment files to become corrupt during disk pressure.
75%A bug or abrupt process termination during shard relocation left partially written or mismatched segment blobs.
60%Misconfigured or insufficient disk space on target nodes causing failures during copy and checksum mismatches.
20%Suggested fixes
2
Diagnostic step 2
If an uncorrupted replica exists on another node, allocate that replica as primary using the reroute API (promote replica) to recover without loss.
3
Diagnostic step 3
If no healthy copy exists, plan to restore the affected indices from the most recent snapshot. Verify snapshot integrity first with a restore-to-different-cluster or 'snapshot' verification where possible.
4
Diagnostic step 4
If node disks show SMART reallocated sectors or IO errors, take the node offline, image the disks for forensics, and replace failing drives—escalate to storage/hardware vendor support if hardware failure is suspected.
5
Diagnostic step 5
If unsure whether corruption is limited to a subset of segment files, consider contacting Elastic support or a specialist for shard-level recovery; avoid running destructive commands that overwrite shards without a verified snapshot backup.