Elasticsearch cluster won't relocate shards: 'disk threshold exceeded' though df -h shows free space (EC2 EBS ext4 reserved blocks suspected)

Environment

Elasticsearch 7.10 cluster on AWS EC2: three data nodes (m5.large equivalents), each node has a single EBS volume mounted at /var/lib/elasticsearch with ext4 filesystem. cluster.routing.allocation.disk.threshold_enabled=true. Nodes run OpenJDK 11 and are started via systemd with path.data=/var/lib/elasticsearch.

Symptoms

Cluster health moved from green → yellow after one node received large bulk indexing. Elasticsearch 'shard allocation' does not relocate shards away from the node flagged as 'high disk usage'; those shards remain unassigned and some indices degrade. Administrators observe cluster won't allocate replicas to other nodes citing 'cannot allocate, disk threshold exceeded'.

Observations

df -h reports /var/lib/elasticsearch with 35% usage and >200GB available on each node. GET _cluster/allocation/explain for an unassigned shard returns 'too many shards on node' and 'disk usage exceeds high watermark' for the targeted node. Elasticsearch logs show disk-based allocation decisions and reference java.io.File.getUsableSpace returning low usable bytes on the affected node. No other processes are consuming obvious additional space; EBS volumes show expected provisioned size. No snapshots running at the observed time.

Already attempted

Tried temporarily lowering cluster.routing.allocation.disk.watermark.high and reran allocation explain — cluster still refused to move some shards. Restarted the node and Elasticsearch service on the affected host (no change). Verified path.data and mount options; checked for bind mounts and symlinks (none found).

Question

Why is Elasticsearch treating a node as over-high-disk even though df -h shows plenty of free space, and what safe checks can I run to determine whether filesystem reserved blocks, Java usable-space semantics, or path.data misconfiguration are the culprit?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

Filesystem reserved blocks/quota (ext4 reserved_blocks) or root-only reserved space reduces 'usable' space visible to the ES process, causing Java File.getUsableSpace to report below watermark 70%

path.data misconfigured or node has multiple mountpoints (bind mounts or symlink) so ES checks a different mount with low usable space 45%

EBS snapshot or backup process temporarily reserved space or left open filehandles visible only to root, reducing usable space as seen by the ES process 30%

Summary

Elasticsearch uses the JVM API (File.getUsableSpace) to compute available space; that API can reflect the filesystem's usable space after ext4 reserved blocks (typically 5%) or mount-level quotas, so 'df -h' showing free space for root vs usable for non-root can differ. On EC2, EBS volumes with ext4 default reserved blocks or mount options (e.g., reserved blocks for root) can make ES see less usable space. Another possibility is that path.data points to a different or duplicated mount, or that Elasticsearch process lacks permission to the mountpoint leading to File.getUsableSpace returning a lower figure. The logs pointing at File.getUsableSpace strongly indicate the 'usable bytes' reported to Java differs from df output.

Suggested Fixes

Diagnostic step 1

Run 'sudo -u elasticsearch java -XshowSettings:properties -version' is not needed; instead use a small Java snippet (or use Python os.statvfs) running as the elasticsearch user to print usable space: e.g., 'sudo -u elasticsearch python3 - <<'PY' import os s=os.statvfs("/var/lib/elasticsearch") print(s.f_bavail*s.f_frsize) PY' This shows the same metric Java would see. Do not modify files yet.

Diagnostic step 2

Check filesystem reserved block percentage: 'sudo tune2fs -l /dev/nvmeXn1 | grep -i 'Reserved block count'' and inspect reserved_block_count * block_size. If reserved blocks explain the delta, you can reduce the reserve (e.g., tune2fs -m 1) but only after maintenance window and ensuring root won't be starved.

Diagnostic step 3

Confirm path.data mount: 'mount | grep /var/lib/elasticsearch' and compare with ES config. If path.data points at a bind mount or symlink, correct the configuration and restart ES in a rolling manner.