KVM guests lose all network after host NIC firmware update under sustained I/O — MTU resets and guests show 1500-only traffic

Environment

Proxmox KVM host (Linux 5.15), Mellanox/Intel dual-port 25Gb NIC; firmware and vendor driver updated to latest on host during maintenance. Guests use virtio-net vNICs bridged to host bond; previously ran with MTU 9000 and stable.

Symptoms

After host firmware/driver update and under sustained disk/network I/O, all KVM guests lose external connectivity (ping fails to external addresses). Guest-to-guest on same host still works. Host can reach network. Observed that guest interfaces drop jumbo frames and negotiate MTU 1500; large TCP transfers stall from guests but small control traffic sometimes works.

Observations

Host dmesg shows NIC driver warnings about 'invalid TSO/GSO offload settings' and occasional VF/VFIO detach logs. ethtool shows host physical NIC still set to MTU 9000, but bridge and guest interfaces show 1500. Reverting to previous firmware on test host not yet attempted. No high CPU or out-of-memory events. IOMMU passthrough not used.

Already attempted

Rolled back network bond to single port and toggled bridge forwarding off/on; guest connectivity briefly restored but degraded under load. Restarting libvirtd and restarting a single guest's network interface restored connectivity for that guest only. Collected host kernel logs, ethtool -k, and guest ip link outputs.

Question

What is the most probable explanation for guests losing jumbo-frame connectivity after NIC firmware/driver update only under load, and how to restore network for all guests without full host rollback or extended downtime?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

NIC firmware/driver regression breaking offload handling (TSO/GSO/GRO) or MTU propagation to bridge/vhost, triggered under high I/O 85%

Host bridge or bond configuration edge-case incompatibility with new driver (e.g., VLAN/jumbo handling bug) 55%

IOMMU or VF reset behavior changed causing transient detach of vhost-net devices under load 30%

Summary

Symptoms (guest MTU collapse to 1500, driver warnings in dmesg, issue triggered under sustained I/O) point strongly to a host NIC firmware or driver regression affecting offloads (TSO/GSO/GRO) or bridge/bridge-netdev interactions. Possible that driver now advertises offload capabilities inconsistently under load or that firmware enforces MTU constraints per PF/VF. Guest-to-guest working indicates vhost/virtio path internal to host remains functional, while external offload/MTU negotiation to the wire fails. Reverting firmware or using driver workarounds is likely required.

Suggested Fixes

Diagnostic step 1

Immediate mitigation: reduce guest MTU to 1500 in a controlled fashion (short downtime) to restore connectivity under current firmware if jumbo is not essential, or move a subset of services to a host with previous firmware.

Diagnostic step 2

Collect extended dmesg, ethtool -k, ethtool -S, and bridge/link stats during reproduction; capture the exact driver version and firmware build. Note the warning strings for vendor TAC.

Diagnostic step 3

On a single test host, try booting with the previous driver module (if available) or rollback firmware in a maintenance window to confirm regression. If rollback not feasible, add kernel module options to disable specific offloads (ethtool -K) to see if disabling TSO/GSO/GRO restores guest MTU propagation.