Elasticsearch shard stuck INITIALIZING after disk-pressure eviction; recovery keeps failing with checksum mismatch

Environment

Self-hosted Elasticsearch 7.10 cluster: 5 data nodes, replication factor 1, daily indexing of time-series logs. One node experienced disk usage >95% triggering shard eviction and subsequent restart of that node.

Symptoms

Cluster health remains yellow; several shards show status INITIALIZING and never reach STARTED. Indexing continues on other shards but searches return partial results for affected indices. Recovery processes repeatedly attempt to copy segments but fail and retry.

Observations

Elasticsearch logs on the recovering node contain repeated entries like 'failed to copy blob ... checksum mismatch' and 'corrupt file detected, failing shard recovery'. The node with prior disk pressure shows IO errors in system dmesg around the time of eviction. 'GET /_cat/shards' shows those shards assigned to the affected node and stuck INITIALIZING. Snapshots exist for daily backups up to 24 hours prior to the event.

Already attempted

Restarted the affected node; increased 'cluster.routing.allocation.node_initial_primaries_recoveries' to limit concurrent recoveries; set 'cluster.routing.allocation.enable' to 'primaries' and back, attempted reroute of shards to other nodes, and used 'allocate_empty_primary' on one shard to bring it up (but that loses data for that shard). Attempted to run 'fsck' at OS level on the node's disks; SMART tests show some non-critical reallocated sectors but no outright failure.

Question

How should I safely recover these stuck INITIALIZING shards that log checksum or corruption errors without risking further data loss, and what diagnostics will confirm whether a restore from snapshot is required versus a local salvageable recovery?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Hardware issues (failing SSD/HDD) producing IO errors and corrupted segments. 90%

Filesystem-level corruption on the node that hosted the primary shard caused segment files to become corrupt during disk pressure. 75%

A bug or abrupt process termination during shard relocation left partially written or mismatched segment blobs. 60%

Summary

Repeated checksum mismatch and corrupt file messages during recovery indicate underlying segment files on the source or target node are damaged or inconsistent. Disk pressure likely caused partial writes or filesystem-level corruption. Options are limited: if corruption affects primary shard files, recovery from replicas or snapshot is required; if replicas are also corrupted, snapshot restore is safest. Using 'allocate_empty_primary' forces recovery but loses the shard's data; salvage tools are risky. It's necessary to determine whether any healthy copies exist on other nodes or in snapshot.

Suggested Fixes

Diagnostic step 1

Do not run further automated reallocation that could overwrite remaining copies. Identify which node currently contains the most up-to-date copy of each affected shard using 'CAT SHARDS' and node logs.

Diagnostic step 2

If an uncorrupted replica exists on another node, allocate that replica as primary using the reroute API (promote replica) to recover without loss.

Diagnostic step 3

If no healthy copy exists, plan to restore the affected indices from the most recent snapshot. Verify snapshot integrity first with a restore-to-different-cluster or 'snapshot' verification where possible.