Kubernetes nodes fail to pull container images after containerd overlay storage filled on GKE-like cluster

Environment

On-prem K8s cluster (v1.26) with containerd 1.7, nodes provisioned from a node pool with 200 GB root disk; workloads include several large images (~2–6 GB each) and a nightly batch job that pulls and extracts images concurrently.

Symptoms

New pods enter ImagePullBackOff or ErrImagePull with errors mentioning 'no space left on device' or 'failed to unpack image' when scheduled to nodes; existing running containers continue but any attempt to start new pods fails on affected nodes.

Observations

df -h shows /var/lib/containerd/overlay2 consuming >95% of disk; journalctl on node shows containerd errors 'failed to reserve layer' and overlayfs lowerdir/upperdir errors; kubelet reports 'image pull rate limited' and Docker registry is reachable (curl succeeds). After cleaning a few images manually, some pulls succeed for a short time before problem returns.

Already attempted

Manually docker/containerd image prune on affected nodes (containerd ctr images rm and runc delete) temporarily freed space and allowed pods to start; increased ephemeral disk size on one node and cordoned/drained it to test (worked on that node); configured image garbage-collection thresholds in kubelet (no immediate effect).

Question

How can I reliably diagnose why overlay storage grows uncontrollably under these workloads, implement safe cleanup that won't disrupt running pods, and put in place limits or automation to prevent recurrence at scale?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Insufficient ephemeral disk allocated to nodes relative to workload image sizes and frequency of redeploys 90%

Slow or disabled image garbage collection in kubelet/containerd leading to accumulation of unused layers 75%

Failed or partial pulls leaving dangling layer data that isn't reclaimed automatically 70%

Summary

The immediate failure mode is disk exhaustion in containerd's storage area (overlay/overlay2), which prevents pulling/unpacking images. Root causes include insufficient image garbage collection, large numbers of image layers/version churn, dangling image layers from failed pulls, or misconfigured ephemeral disk size. Long term fixes include tuning kubelet/containerd GC, increasing node storage, and avoiding large image churn. Careful cleanup is required to not affect running containers.

Suggested Fixes

Diagnostic step 1

Immediately cordon and drain a test node and perform containerd image garbage collection: stop kubelet, use 'containerd-ctr images ls' and 'ctr content' to identify large unused blobs, then remove unused images; restart kubelet. Collect logs before and after.

Diagnostic step 2

Enable kubelet image garbage collection and tune high/low thresholds (imageGCHighThresholdPercent, imageGCLowThresholdPercent) across the node pool; roll out as a test on a small subset first.

Diagnostic step 3

Implement monitoring/alerting on usage of /var/lib/containerd and overlay storage (node-exporter + Prometheus node_disk_* and containerd metrics) to catch growth before exhaustion.