KVM guest interface MTU drops to 1500 and large TCP transfers stall after enabling 802.1Q VLAN tagging on host bridge

Environment

KVM host (Ubuntu 22.04) using libvirt-managed bridges (br0). Host NIC is 10 GbE with MTU 9000. After enabling VLAN trunking on br0 and adding VLAN subinterfaces (br0.100), multiple KVM guests with virtio-net interfaces report degraded large-transfer throughput and packet loss. Guests run Debian 11 with MTU 9000 configured.

Symptoms

Large file copies over the guest-to-guest and guest-to-storage TCP connections show severe throughput drop and stalled transfers; small packets remain responsive. Guests report occasional 'network is unreachable' on certain flows and Re-transmits spike. Host networking shows no full interface errors on br0 but br0.100 shows some dropped packets counters incrementing.

Observations

ip link show reports guest vnet interfaces with mtu 9000. tcpdump on host physical NIC shows VLAN-tagged frames with correct VLAN IDs but some large frames being fragmented or dropped (seen as incomplete frame sequences). ethtool -S on the physical NIC shows hardware offload counters for VLAN‑tagging and GRO/TSO enabled; when VLAN tagging is performed in software (on the host) large frames traverse without issue. syslog contains occasional kernel message: 'netdev: failed to send packet due to XDP/driver constraints' during high throughput. No errors at the switch port level; switch configured for trunking with MTU 9000.

Already attempted

Disabled guest offloads (ethtool -K on guest vnet0: tso off gso off gro off) and re-tested — throughput improved slightly but stalls still occur. Temporarily removed VLAN tagging on the host and saw traffic resume at expected rates. Upgraded host kernel to latest 5.15 LTS backport and libvirt to latest stable; problem persisted. Rebooted host and guests.

Question

What diagnostics can pinpoint whether the issue is (a) host NIC driver VLAN offload bug, (b) qemu/virtio interaction with VLAN-tagged bridges and offloading, or (c) switch / trunk MTU mismatch? Which reversible settings should I change first to isolate without disrupting production guests, and when is it appropriate to escalate to NIC vendor or upstream kernel bug tracker?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Host NIC driver incorrectly handling VLAN offloads with TSO/GSO enabled, producing bad hardware descriptors for jumbo frames. 90%

Kernel/firmware bug in NIC offload path that manifests only when VLANs and jumbo frames are combined under high throughput. 80%

qemu/virtio and linux bridge interaction with VLAN-tagged bridges leading to double-tagging or mis-sized frames that the NIC rejects. 35%

Summary

Behavior strongly suggests interaction between VLAN offloading, jumbo frames, and virtio/bridge handling on the host. The fact that removing VLAN tagging from the host resolves the issue and that kernel driver logs reference send failures during high throughput indicates a host-side driver/firmware or offload path bug rather than guest configuration. Offloads (TSO/GSO/GRO) can cause the host to hand oversized skbs to the NIC which, when combined with VLAN tag processing in hardware, can exceed device constraints and result in drops or send failures.

Suggested Fixes

Diagnostic step 1

On a single non-critical guest, disable virtio offloads (ethtool -K vnet0 tso off gso off gro off) and test large transfers to confirm impact; revert after test.

Diagnostic step 2

Temporarily disable VLAN offload on the host NIC (ethtool -K eth0 rxvlan off txvlan off) and test with bridge VLANs active; if transfers recover, offload is implicated.

Diagnostic step 3

Run tcpdump on host physical interface to verify whether frames leaving the NIC are correctly VLAN-tagged and within MTU; capture both host and guest to compare.