1
Diagnostic step 1
On affected node, check actual inode/socket: ls -l /var/lib/kubelet/plugins/<driver>/ and compare ownership/permissions to a working node. Do not restart storage pods yet.
FULL AI ANALYSIS
Kubernetes PVCs stuck Pending after node migration — CSI node plugin fails to register with kubelet
Generated Oct 13, 2026, 9:45:52 AM
Analysis summary
CSI node registration relies on a well-known unix domain socket under /var/lib/kubelet/plugins//controller or /node. Failures commonly come from permission/ownership mismatches, kubelet and CSI plugin running with different user namespaces or container runtimes not mounting the host path into the plugin, or socket path length/SELinux/AppArmor confinement. The recent change in container runtime and cgroup driver suggests the node plugin container may not have hostPath mounts configured correctly or kubelet changed socket ownership, preventing registration.
Possible causes
DaemonSet manifest uses hostPath with subPath and the container runtime changes stopped creating the socket due to mount propagation differences.
80%kubelet now runs with different user/group resulting in permission mismatch on /var/lib/kubelet/plugins so plugin cannot create or kubelet cannot access the socket.
75%Container runtime overlaying /var/lib/kubelet (different mount namespace) so plugin's hostPath mount is empty inside the container.
70%AppArmor or seccomp profile applied to CSI plugin container blocks creation/access to unix domain sockets under /var/lib.
5%Suggested fixes
2
Diagnostic step 2
Inspect kubelet logs: journalctl -u kubelet -o cat | grep -i csi and check for 'failed to register' entries; capture first occurrence timestamp.
3
Diagnostic step 3
Check DaemonSet YAML used for CSI node plugin: confirm hostPath mounts include the exact /var/lib/kubelet/plugins path and that mountPropagation is set to 'Bidirectional' if required by driver. If missing, apply fix and rollout DaemonSet update.
4
Diagnostic step 4
If a permission mismatch exists, temporarily grant broader permissions (chown root:root and chmod 0755) to the plugin directory to test registration; do this carefully and revert if not needed.
5
Diagnostic step 5
If probe suggests container runtime mount isolation, test running a debug container with the same hostPath mounts to confirm socket visibility inside container. Escalate to storage vendor support if driver binary logs show internal errors; avoid remounting or deleting PV data directories to prevent data loss.