1
Diagnostic step 1
Collect detailed kubelet and CSI logs around a failing mount (kubelet -v=4 and CSI nodeplugin logs) and correlate timestamps with kernel dmesg and udevadm monitor events to confirm event order.
FULL AI ANALYSIS
Kubernetes StatefulSet pods fail to mount Ceph PVCs on nodes upgraded to newer kubelet: 'failed to find device' in kubelet logs
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
This behavior commonly stems from a timing/race between CSI attach reporting and the node's device path being created (udev), or changes in kubelet's attach/detach or mounter discovery logic between versions. If the device node appears after kubelet checks for it (or under a different path that kubelet doesn't expect), kubelet will error. Other possibilities include stricter device path validation in the newer kubelet, changes in kernel udev event ordering, or default switching between in-tree and CSI attachment flows.
Possible causes
Race between CSI attach success being reported and the kernel/udev creating the block device node (udev event latency)
95%kubelet version change tightened device discovery semantics or path expectations
70%CSI driver or node plugin binary incompatibility with the newer kubelet leading to a mismatched attach/detach contract
20%Incompatible or outdated udev rules on upgraded nodes that result in different device naming/permissions
10%Suggested fixes
2
Diagnostic step 2
On an affected node, reproduce a failing mount and run 'udevadm monitor' and 'lsblk -o NAME,KNAME,MAJ:MIN' to capture when the device node appears versus when kubelet attempts mount.
3
Diagnostic step 3
Temporarily increase kubelet mount/device discovery timeouts (mountPropagation/mountTimeout options if available) or set the CSI driver to use NodePublish staging (if supported) to mitigate race windows.
4
Diagnostic step 4
Ensure the rbd kernel module and user-space helpers are the same versions across nodes and that udev rules are consistent; consider adding a short udev settle/wait hook in the CSI node plugin as a reversible mitigation.
5
Diagnostic step 5
If diagnostics point to a kubelet/CSI contract regression, reproduce on a staging node with a prior kubelet version to confirm, then escalate to the CSI driver maintainers or Kubernetes SIG-Storage with logs and timelines for further investigation.