ESXi VM fails to resume from snapshot consolidation on NFS datastore after upgrading multipath driver

Environment

VMware ESXi 7.0 hosts (3-node cluster) using an NFSv4.1 datastore exported by a Linux storage appliance running custom multipath driver v3.2. Recently upgraded hosts to ESXi build 19482537 and applied vendor multipath module update on storage appliance. A critical VM on the NFS datastore fails to resume after snapshot consolidation with errors.

Symptoms

vSphere reports 'Failed to resume virtual machine' after snapshot consolidation operation. VM remains in 'suspended' state; attempting to power on results in 'File handle not found' and 'Cannot open the disk' errors. Other VMs on the same host and datastore are unaffected. Host logs show I/O errors to the specific VM's VMDK path during consolidation.

Observations

Hostd and vmkernel logs show 'NFS: Handle not found' and stale file handle traces when ESXi attempts to reopen the consolidated disk. On the storage appliance, dmesg shows NFS server side error 'fh invalidated' and multipath logs report path revalidation and path IDs changing during snapshot consolidation. Rolling back the storage multipath driver to previous version temporarily restores ability to open the VMDK (but risk). No network partitions observed; NFS server uptime stable.

Already attempted

Restarted hostd and vpxa services on ESXi host (no effect). Attempted to re-register VM from datastore — registration fails with handle errors. Performed storage rescan and cleared multipath caches on storage appliance, allowing some VMs to recover but the consolidated VM still failed. Restoring a backup snapshot and re-importing succeeded on an alternate datastore.

Question

Why does the VM fail to resume after snapshot consolidation with 'file handle not found' after the multipath driver update, and what reversible diagnostics and recovery steps are safe to pursue before rolling back storage driver or restoring from backups?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

Storage multipath driver update on the NFS server invalidates or rotates file handles during path revalidation, breaking ESXi's open file descriptors mid-consolidation. 95%

Race condition in consolidation I/O pattern that exposes bug in NFS server multipath or ESXi NFS client handling of file handle changes. 80%

NFS server implementation incompatibility with ESXi build after driver update (e.g., changes to export fsid or inode handling). 35%

Summary

Snapshot consolidation involves writing new VMDK data while the NFS server maintains file handles. The storage multipath driver update appears to change path identifiers or trigger server-side file handle invalidation during consolidation, causing ESXi to lose reference to the consolidated VMDK. This leads to stale file handle errors and failure to reopen disks. The fact other VMs are unaffected suggests the issue triggers under specific timing/IO patterns (consolidation writes) for certain files.

Suggested Fixes

Diagnostic step 1

Collect synchronized logs: ESXi vmkernel and hostd logs during a controlled consolidation attempt, plus storage appliance syslog and NFS debug logs to identify exact moment of file handle invalidation.

Diagnostic step 2

Reproduce on a non-production VM: create a similar snapshot chain and perform consolidation while tracing NFS file handle lifecycle to confirm reproducible behavior.

Diagnostic step 3

Temporarily avoid consolidating VMs during peak activity and schedule maintenance window to test fixes. Do not force-fail or delete files without backups.