1
Diagnostic step 1
Immediately stop destructive workloads and ensure a verified, recent backup of all data on the NVMe drive. Do not continue stress testing on production data.
FULL AI ANALYSIS
NVMe SSD Reallocated Sector Count increases only during heavy random-write FIO workload on Linux
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
The pattern—errors and SMART reallocation only under heavy random writes—points to either (a) the drive's internal wear-leveling, garbage collection, or mapping tables being stressed by random writes and forced remapping when blocks fail, or (b) a drive firmware bug or internal controller issue that manifests only under sustained high IOPS/randomness. The repeated I/O errors in kernel logs and reproduction across hosts rule out a single host/slot issue, making a drive-internal fault or firmware/firmware‑+hardware interaction the most likely root cause. Given SMART attribute growth, data integrity risk increases and backing up is essential.
Possible causes
NAND flash cells encountering write/erase failures under sustained random write patterns, causing remapping and SMART attribute increases (endurance/wear-related).
75%Drive firmware bug handling high IOPS or heavy random write patterns triggers inappropriate remapping or error handling (seen as intermittent I/O errors).
65%Internal controller thermal throttling leading to elevated error retries and remap operations under stress, despite ambient temperature measured OK (internal hotspot).
40%Power delivery or PCIe link issues causing transient write failures only under sustained high traffic (less likely given reproduction on different hosts).
15%Suggested fixes
2
Diagnostic step 2
Collect diagnostic evidence: capture full smartctl -a output before and after a controlled stress test, save dmesg logs showing blk_update_request I/O errors, and run nvme-cli 'nvme list' and 'nvme smart-log' to gather vendor lifetime and error counts. These artifacts are necessary for vendor RMA evaluation.
3
Diagnostic step 3
Run a controlled stress test on a non-production copy (or a low-value partition) to reproduce the issue while recording kernel logs; prefer incremental runs (lower iodepth) to determine threshold IOPS that trigger reallocations.
4
Diagnostic step 4
If vendor-provided firmware updates exist, consult vendor support before applying—if advised, update firmware and repeat controlled tests. If vendor tools decline update or issue persists after firmware update, open an RMA with the drive vendor, supplying collected smartctl and kernel logs. Do not attempt low-level remapping utilities that may obscure evidence required for RMA.
5
Diagnostic step 5
As a temporary mitigation for important services, move critical workloads to a different SSD and mark this drive as suspect. Escalate to vendor support promptly for replacement if diagnostics reproduce across hosts or after a vendor-recommended firmware update fails to resolve the issue.