NVMe SSD sustained write throughput collapses under Linux when PCIe ASPM is enabled

Environment

Desktop motherboard X570 with AMD CPU, Ubuntu 22.04 running kernel 5.15. NVMe Samsung 980 Pro in primary M.2 slot. BIOS default power settings enabled ASPM/L1; kernel parameters unmodified by default.

Symptoms

Sustained sequential write (fio, 1 MiB block, direct=1) starts at expected ~2.8 GB/s but drops to ~400–600 MB/s after ~20–40 seconds when ASPM/L1 is enabled. I/O latency spikes correspond with throughput drops. Disabling ASPM in BIOS or booting with pcie_aspm=off restores sustained throughput.

Observations

dmesg reports frequent PCIe link state changes and NVMe controller power state transitions around the time throughput drops. lspci -vv for the NVMe device shows L1 supported and active; nvme-cli shows SMART read/write counters OK and no media errors. Firmware update for the drive applied (latest available). Changing fio concurrency and queue depth affects initial throughput but not the eventual drop. Swapping the drive to a different slot produced the same behavior when ASPM was enabled.

Already attempted

Updated NVMe firmware and motherboard BIOS to latest recommended by vendor, tried kernel boot params pcie_aspm=force and pcie_aspm=off, tested with different ioengine and block sizes, swapped to a PCIe adapter card and retested (similar issue when ASPM allowed). Confirmed disabling ASPM restores sustained performance. Collected dmesg and lspci outputs for analysis.

Question

Is the throughput collapse most likely caused by a device/BIOS/driver ASPM interaction bug, and what is the recommended setting to balance power savings and NVMe performance safely in production? Also what diagnostic data should be collected before filing bug reports with vendors?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

NVMe controller or drive firmware implements ASPM/L1 incorrectly, causing frequent link retrain or excessive wake latencies. 80%

Motherboard/BIOS aggressiveness in enabling L1/L0s or L1SS features the drive does not fully support, producing instability. 65%

Kernel PCIe power management bug or missing platform quirk leading to incorrect negotiation of link power states. 55%

Summary

ASPM (Active State Power Management) reduces PCIe link power by transitioning into low-power link states (L0s/L1). Some NVMe controllers, host chipsets, or BIOS implementations have buggy ASPM/L1 or L1 Substates (L1SS) behavior that causes frequent link retraining or adds large wake-latency, which drastically reduces sustained throughput. The fact throughput returns when ASPM is disabled indicates a link power management interaction rather than a pure NVMe media issue. Fixes may require BIOS/firmware/kernel quirk updates from vendor.

Suggested Fixes

Diagnostic step 1

Reproduce and document the issue reliably: run fio with logging while toggling ASPM (BIOS vs pcie_aspm=off/force) and capture dmesg, lspci -vv, and nvme-cli get-log outputs showing timestamps of link state changes and throughput drop. These artifacts are required when filing vendor bugs.

Diagnostic step 2

If sustained throughput is required for your workload, keep ASPM disabled (BIOS or kernel parameter pcie_aspm=off) — this is reversible and the safest short-term mitigation. Note the power trade-off (higher idle power draw) and document which systems need this profile.

Diagnostic step 3

Test with alternate BIOS settings that selectively disable L1/L0s/L1SS rather than blanket ASPM off to find the minimal change that restores throughput; try disabling L1SS first if BIOS exposes it.