NVMe SSD Reallocated Sector Count increases only during heavy random-write FIO workload on Linux

Environment

Desktop workstation with NVMe SSD in PCIe Gen3 x4 slot, Linux kernel 5.15, running fio random-write workload (randwrite 4k, iodepth 32) targeted at a single large file on an ext4 partition; SMART data polled via smartctl.

Symptoms

SMART attribute 5 (Reallocated Sector Count) and attribute 197 (Current Pending Sector) increase by a small amount only when the fio random-write test runs; values remain stable under light sequential writes and normal desktop use. Performance under the test shows occasional write latency spikes and retry counts increase.

Observations

SMART readouts before and after test show consistent growth in attribute 5 only after the heavy random-write test. dmesg contains intermittent 'blk_update_request: I/O error' entries associated with the NVMe device during peak workload windows. Running the same fio test at a lower IO depth or with sequential patterns does not reproduce the reallocated sector increments. Moving the drive to a different PCIe slot and repeating the workload produced the same SMART behavior. Ambient temperatures measured during the test were within vendor-specified range.

Already attempted

Backed up critical data, ran vendor NVMe diagnostic (read-only health check) which reported 'SMART OK' but did not exercise heavy random write patterns, downgraded fio I/O depth and observed reduced incidence of errors, ran badblocks in read-only mode (no errors), updated NVMe firmware via vendor tool where available (firmware version unchanged by vendor tool), and tested the drive in another Linux machine which reproduced the behavior under identical random-write stress.

Question

What are the likely root causes of NVMe sector reallocation appearing only under heavy random-write workloads (rather than during normal use), which reversible diagnostics can confirm whether this is NAND wear leveling, a firmware remapping bug, or transient PCIe errors, and what immediate mitigations should be taken to protect data before pursuing RMA?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

NAND flash cells encountering write/erase failures under sustained random write patterns, causing remapping and SMART attribute increases (endurance/wear-related). 75%

Drive firmware bug handling high IOPS or heavy random write patterns triggers inappropriate remapping or error handling (seen as intermittent I/O errors). 65%

Internal controller thermal throttling leading to elevated error retries and remap operations under stress, despite ambient temperature measured OK (internal hotspot). 40%

Summary

The pattern—errors and SMART reallocation only under heavy random writes—points to either (a) the drive's internal wear-leveling, garbage collection, or mapping tables being stressed by random writes and forced remapping when blocks fail, or (b) a drive firmware bug or internal controller issue that manifests only under sustained high IOPS/randomness. The repeated I/O errors in kernel logs and reproduction across hosts rule out a single host/slot issue, making a drive-internal fault or firmware/firmware‑+hardware interaction the most likely root cause. Given SMART attribute growth, data integrity risk increases and backing up is essential.

Suggested Fixes

Diagnostic step 1

Immediately stop destructive workloads and ensure a verified, recent backup of all data on the NVMe drive. Do not continue stress testing on production data.

Diagnostic step 2

Collect diagnostic evidence: capture full smartctl -a output before and after a controlled stress test, save dmesg logs showing blk_update_request I/O errors, and run nvme-cli 'nvme list' and 'nvme smart-log' to gather vendor lifetime and error counts. These artifacts are necessary for vendor RMA evaluation.

Diagnostic step 3

Run a controlled stress test on a non-production copy (or a low-value partition) to reproduce the issue while recording kernel logs; prefer incremental runs (lower iodepth) to determine threshold IOPS that trigger reallocations.