Proxmox live migration fails when VM has VFIO‑passed PCIe GPU attached (error: device busy)

Environment

Proxmox VE 7.3 cluster with two nodes (QEMU/KVM), VMs running Linux guests. A VM has a PCIe GPU passed through using vfio-pci (bind via resident host), no snapshots, storage on shared NFS. Live migration (qm migrate) fails with error 'device is busy' only for VMs with VFIO devices attached; normal VMs without passthrough migrate fine.

Symptoms

Attempted live migration of VM with GPU attached returns: 'migration failed: device is busy' after several seconds. VM remains running on source node; no crash. Cold migration (stop VM then migrate) succeeds. VM consoles remain accessible on source. This occurs consistently for the same VM and same GPU device; other VMs without passthrough not affected.

Observations

Host logs (journalctl and /var/log/syslog) show errors around vfio stating 'IOMMU group in use' and 'device cannot be detached from host' during migration attempt. - QEMU process holds open file descriptors for /dev/vfio/* and sysfs PCI device nodes. - IOMMU group includes additional devices (e.g., USB controller) preventing isolated migration of only GPU unless they are also passed. - Hardware is older motherboard with limited ACS grouping; ACS override not available or not recommended. - SR-IOV not relevant; device is a consumer GPU that does not support live migration of physical device between hosts.

Already attempted

Tried online migration with --with-local-disks off (storage shared) and same error. - Attempted to unbind vfio-pci from VM guest and rebind on host during live migration (unsafe while VM running) — not attempted further. - Confirmed VM uses QEMU 6.2 as packaged in Proxmox; verified that cold migration works. - Reviewed Proxmox documentation: passthrough devices are typically incompatible with live migration unless mediated devices or device replication techniques used.

Question

Why does live migration fail for VMs with a VFIO-passthrough PCIe GPU attached and what safe, reversible options exist to enable migration or minimize downtime when moving such VMs between Proxmox nodes?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Using true PCIe passthrough of a physical GPU which has no live-migration support (no vGPU or mediated device driver) 95%

Device in same IOMMU group as other host devices, preventing isolated reassignment or hot-detach 90%

Kernel or driver holds the device open on the source host preventing QEMU from migrating the VM with an attached device 85%

Summary

Physical PCIe device passthrough with vfio binds a device to a single host and exposes direct hardware to the guest; most physical devices cannot be live-migrated because the device state and DMA mappings live on the source host and there is no standard mechanism to transfer them atomically to the target. IOMMU group boundaries and device ownership further prevent detaching the device. Consumer GPUs typically lack mediated device support required for safe live migration. Thus Proxmox/ QEMU reports 'device busy' because the device cannot be detached from the host for migration.

Suggested Fixes

Diagnostic step 1

If live migration is required, replace direct passthrough with a migratable alternative: use vendor mediated devices (vGPU / MDEV) supported by the GPU vendor (NVIDIA GRID, Intel GVT-g) and ensure guest/host drivers support migration — test on non-production first

Diagnostic step 2

Use cold migration as an operational workaround: schedule short maintenance window, shut down VM, migrate, and start on destination (reversible, simplest approach)

Diagnostic step 3

Consider using PCI device replication techniques: move workloads to virtualized GPUs or cloud GPUs instead of direct passthrough to enable mobility