1
Diagnostic step 1
Confirm cgroup drivers: check kubelet --cgroup-driver and containerd config (sudo cat /etc/containerd/config.toml) and ensure both use 'systemd' or 'cgroupfs' consistently; avoid changing in production without testing
FULL AI ANALYSIS
Kubernetes pods repeatedly OOM-killed even though node has ample free memory — cgroup vs kubelet limit mismatch
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
The kernel OOM killer is targeting container processes because the container's cgroup memory limit is lower than expected. When pods omit memory limits, the container runtime may still apply default cgroup limits derived from node or kubelet configuration (e.g., kubelet's --cgroup-root or a systemd slice with a small Limit). A common root cause is misconfiguration between kubelet and container runtime cgroup drivers (systemd vs cgroupfs), or a systemd slice with low memory.limit_in_bytes applied to the container's parent slice. Another possibility is node-level memory enforcement via a memory balloon or VM-level limits that reduce available memory for cgroups. The fact explicit limits change behavior indicates cgroup controls are effective and current defaults are too low.
Possible causes
Systemd slice units (e.g., docker.service or system.slice) have MemoryMax set low, causing child container cgroups to inherit tight limits
85%CRI (containerd) or a node bootstrap script sets default runtime memory limit for containers when no limit specified
75%Kernel memory accounting (CONFIG_MEMCG) or swap accounting disabled causing kubelet to fall back to systemd-enforced limits
70%Mismatch between kubelet cgroup driver and container runtime leading to limits applied at different cgroup hierarchies
30%Suggested fixes
2
Diagnostic step 2
Inspect effective cgroup limits for a live container: identify container PID, then check memory.max and memory.usage_in_bytes under /sys/fs/cgroup/<slice>/... and compare to expected values
3
Diagnostic step 3
Check systemd slice configs for MemoryMax: systemctl show --property=MemoryAccounting yes and inspect /etc/systemd/system/*.slice for MemoryMax entries; if present and inappropriate, test removing or raising them and restart services (reversible by restoring original files)
4
Diagnostic step 4
If containers rely on default limits, add explicit memory.limits in pod spec as an immediate mitigation to prevent host OOM kills; use HorizontalPodAutoscaler and proper requests/limits in deployment manifests
5
Diagnostic step 5
Collect node and kubelet logs, and if cgroup driver mismatch or systemd enforced limits are found and cannot be safely changed, escalate to cluster infra team to coordinate configuration changes and plan a rolling node update (changes here can affect many workloads)