Kubernetes PVCs stuck Pending after node migration — CSI node plugin fails to register with kubelet

Environment

On-prem K8s 1.25 cluster running kubelet systemd service on Ubuntu 22.04 nodes. CSI driver for vendor SAN is deployed as a DaemonSet (node plugin) and StatefulSet (controller). Problem observed after rolling nodes to a new image with a different container runtime and cgroup driver (systemd -> cgroupfs).

Symptoms

New nodes show PVCs for StatefulSet pods stuck in Pending with events 'no persistent volumes available for claim'. kubectl get csidrivers shows driver present, but describing the node shows 'no CSINode info' for the driver. Existing pods on old nodes with same storage continue working.

Observations

DaemonSet pods for CSI node plugin are Running on new nodes, but csi plugin logs show repeated 'failed to register node plugin: rpc error: code = Unavailable'. kubelet logs contain 'GRPC: failed to dial unix...' for the csi socket path. /var/lib/kubelet/plugins contains the driver directory but with socket absent or with root-owned socket when plugin runs as non-root.

Already attempted

Restarted CSI node plugin DaemonSet pods and kubelet on affected nodes. Verified SELinux/AppArmor not in Enforcing mode. Compared permissions of /var/lib/kubelet/plugins/* between working and broken nodes; manually created the expected socket file (test) but kubelet reports same error. Did not roll back node image.

Question

What are the likely causes preventing CSI node plugin registration after the container runtime/cgroup driver change, and what reversible diagnostics and fixes can restore PV/PVC binding without data loss?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

DaemonSet manifest uses hostPath with subPath and the container runtime changes stopped creating the socket due to mount propagation differences. 80%

kubelet now runs with different user/group resulting in permission mismatch on /var/lib/kubelet/plugins so plugin cannot create or kubelet cannot access the socket. 75%

Container runtime overlaying /var/lib/kubelet (different mount namespace) so plugin's hostPath mount is empty inside the container. 70%

Summary

CSI node registration relies on a well-known unix domain socket under /var/lib/kubelet/plugins//controller or /node. Failures commonly come from permission/ownership mismatches, kubelet and CSI plugin running with different user namespaces or container runtimes not mounting the host path into the plugin, or socket path length/SELinux/AppArmor confinement. The recent change in container runtime and cgroup driver suggests the node plugin container may not have hostPath mounts configured correctly or kubelet changed socket ownership, preventing registration.

Suggested Fixes

Diagnostic step 1

On affected node, check actual inode/socket: ls -l /var/lib/kubelet/plugins/<driver>/ and compare ownership/permissions to a working node. Do not restart storage pods yet.

Diagnostic step 2

Inspect kubelet logs: journalctl -u kubelet -o cat | grep -i csi and check for 'failed to register' entries; capture first occurrence timestamp.

Diagnostic step 3

Check DaemonSet YAML used for CSI node plugin: confirm hostPath mounts include the exact /var/lib/kubelet/plugins path and that mountPropagation is set to 'Bidirectional' if required by driver. If missing, apply fix and rollout DaemonSet update.