Kubernetes StatefulSet pods fail to mount Ceph PVCs on nodes upgraded to newer kubelet: 'failed to find device' in kubelet logs

Environment

On-prem kubeadm cluster: control plane on v1.26, worker nodes were recently upgraded from kubelet v1.25 to v1.27. Storage: Ceph RBD provisioned via CSI driver v3.x; nodes use kernel rbd module and rbd-nbd helpers. StatefulSet with ReadWriteOnce PVCs scheduled across nodes.

Symptoms

After the kubelet upgrade, StatefulSet pods scheduled to upgraded nodes fail to start due to mount errors. The same PVCs mount fine on older nodes that haven't been upgraded. Users see pods CrashLoopBackOff with events mentioning MountVolume.MountDevice and logs show 'failed to find device with major:minor' or 'device not found' errors.

Observations

On affected nodes, 'rbd map' executed manually maps the RBD image and the device node appears (/dev/rbdX). The CSI controller reports successful attach. kubelet logs (kubelet -v=4) show the attach success but subsequent mount fails because kubelet cannot find the device path it expects. There are no disk pressure or permission errors. udev events show the device appearing slightly later than the CSI attach completion timestamp.

Already attempted

Restarted kubelet and CSI driver pods, reloaded rbd kernel module, compared udev rules between working and non-working nodes, increased kubelet volume mount retry counts, and drained then uncordoned nodes. Manually mapping and binding the device allows the pod to start, indicating the device is present but not discovered by kubelet in time.

Question

Why does the upgraded kubelet fail to detect device nodes for attached Ceph RBD volumes (even though the device appears), and what reversible changes to kubelet/CSI/udev configuration can make mounts reliable again?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Race between CSI attach success being reported and the kernel/udev creating the block device node (udev event latency) 95%

kubelet version change tightened device discovery semantics or path expectations 70%

CSI driver or node plugin binary incompatibility with the newer kubelet leading to a mismatched attach/detach contract 20%

Summary

This behavior commonly stems from a timing/race between CSI attach reporting and the node's device path being created (udev), or changes in kubelet's attach/detach or mounter discovery logic between versions. If the device node appears after kubelet checks for it (or under a different path that kubelet doesn't expect), kubelet will error. Other possibilities include stricter device path validation in the newer kubelet, changes in kernel udev event ordering, or default switching between in-tree and CSI attachment flows.

Suggested Fixes

Diagnostic step 1

Collect detailed kubelet and CSI logs around a failing mount (kubelet -v=4 and CSI nodeplugin logs) and correlate timestamps with kernel dmesg and udevadm monitor events to confirm event order.

Diagnostic step 2

On an affected node, reproduce a failing mount and run 'udevadm monitor' and 'lsblk -o NAME,KNAME,MAJ:MIN' to capture when the device node appears versus when kubelet attempts mount.

Diagnostic step 3

Temporarily increase kubelet mount/device discovery timeouts (mountPropagation/mountTimeout options if available) or set the CSI driver to use NodePublish staging (if supported) to mitigate race windows.