FULL AI ANALYSIS

Elasticsearch cluster won't relocate shards: 'disk threshold exceeded' though df -h shows free space (EC2 EBS ext4 reserved blocks suspected)

Generated Oct 14, 2026, 9:39:02 AM

Analysis summary

Elasticsearch uses the JVM API (File.getUsableSpace) to compute available space; that API can reflect the filesystem's usable space after ext4 reserved blocks (typically 5%) or mount-level quotas, so 'df -h' showing free space for root vs usable for non-root can differ. On EC2, EBS volumes with ext4 default reserved blocks or mount options (e.g., reserved blocks for root) can make ES see less usable space. Another possibility is that path.data points to a different or duplicated mount, or that Elasticsearch process lacks permission to the mountpoint leading to File.getUsableSpace returning a lower figure. The logs pointing at File.getUsableSpace strongly indicate the 'usable bytes' reported to Java differs from df output.

Possible causes

Filesystem reserved blocks/quota (ext4 reserved_blocks) or root-only reserved space reduces 'usable' space visible to the ES process, causing Java File.getUsableSpace to report below watermark

70%

path.data misconfigured or node has multiple mountpoints (bind mounts or symlink) so ES checks a different mount with low usable space

45%

EBS snapshot or backup process temporarily reserved space or left open filehandles visible only to root, reducing usable space as seen by the ES process

30%

Bug or regression in Elasticsearch provider/JVM on this OS causing misreporting of available disk space

20%

Suggested fixes

Diagnostic step 1

Run 'sudo -u elasticsearch java -XshowSettings:properties -version' is not needed; instead use a small Java snippet (or use Python os.statvfs) running as the elasticsearch user to print usable space: e.g., 'sudo -u elasticsearch python3 - <<'PY' import os s=os.statvfs("/var/lib/elasticsearch") print(s.f_bavail*s.f_frsize) PY' This shows the same metric Java would see. Do not modify files yet.

Diagnostic step 2

Check filesystem reserved block percentage: 'sudo tune2fs -l /dev/nvmeXn1 | grep -i 'Reserved block count'' and inspect reserved_block_count * block_size. If reserved blocks explain the delta, you can reduce the reserve (e.g., tune2fs -m 1) but only after maintenance window and ensuring root won't be starved.

Diagnostic step 3

Confirm path.data mount: 'mount | grep /var/lib/elasticsearch' and compare with ES config. If path.data points at a bind mount or symlink, correct the configuration and restart ES in a rolling manner.

Diagnostic step 4

Use Elasticsearch APIs to explain allocation: GET _cluster/allocation/explain and GET _cat/allocation?v to see the bytes ES uses for decisioning. Avoid changing cluster.watermark settings in production long-term; use them temporarily to allow relocation if you understand consequences.

Diagnostic step 5

If these steps do not reconcile the numbers or changes require filesystem reformatting or significant disk shrink/resize, escalate to infrastructure/storage team and consider opening an Elasticsearch support ticket; provide the cluster logs and outputs of the 'usable space' checks.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads