Login or Register
FULL AI ANALYSIS

Kubernetes nodes fail to pull container images after containerd overlay storage filled on GKE-like cluster

Generated Oct 3, 2026, 12:35:03 AM

Analysis summary

The immediate failure mode is disk exhaustion in containerd's storage area (overlay/overlay2), which prevents pulling/unpacking images. Root causes include insufficient image garbage collection, large numbers of image layers/version churn, dangling image layers from failed pulls, or misconfigured ephemeral disk size. Long term fixes include tuning kubelet/containerd GC, increasing node storage, and avoiding large image churn. Careful cleanup is required to not affect running containers.

Possible causes

Insufficient ephemeral disk allocated to nodes relative to workload image sizes and frequency of redeploys

90%

Slow or disabled image garbage collection in kubelet/containerd leading to accumulation of unused layers

75%

Failed or partial pulls leaving dangling layer data that isn't reclaimed automatically

70%

Misconfigured logging or application data writing into containerd storage paths

20%

Suggested fixes

1

Diagnostic step 1

Immediately cordon and drain a test node and perform containerd image garbage collection: stop kubelet, use 'containerd-ctr images ls' and 'ctr content' to identify large unused blobs, then remove unused images; restart kubelet. Collect logs before and after.

2

Diagnostic step 2

Enable kubelet image garbage collection and tune high/low thresholds (imageGCHighThresholdPercent, imageGCLowThresholdPercent) across the node pool; roll out as a test on a small subset first.

3

Diagnostic step 3

Implement monitoring/alerting on usage of /var/lib/containerd and overlay storage (node-exporter + Prometheus node_disk_* and containerd metrics) to catch growth before exhaustion.

4

Diagnostic step 4

Consider increasing node root/ephemeral disk or using a separate larger disk for containerd storage (mount /var/lib/containerd on dedicated PV) and validate via controlled drain/replace.

5

Diagnostic step 5

Audit CI/CD to reduce image churn: use smaller base images, multi-stage builds, and a retention policy on registry tags to reduce unique layer count; schedule periodic safe node restarts with drain to force reclaiming resources.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads