Kubernetes pods repeatedly OOM-killed even though node has ample free memory — cgroup vs kubelet limit mismatch

Environment

EKS-like on-prem Kubernetes cluster v1.24 running kubelet 1.24, containerd runtime, nodes are 64GB RAM Linux VMs (Ubuntu 22.04) with kube-reserved 1Gi and system-reserved 1Gi configured. A microservice (Go binary) requests 512Mi and sets no memory limit; some pods get restarted by OOM-killer shortly after start although host node free memory remains >10Gi.

Symptoms

Pods show status 'OOMKilled' in kubectl describe and kubelet events show 'Killing container with id ...: container had OOMKilled'. Pods restart repeatedly (CrashLoop). Node-level memory pressure not reported (kubectl top node shows <60% usage). Other pods on same node unaffected.

Observations

dmesg on node shows 'Out of memory: Kill process <pid> (myapp) score 1000 or sacrifice child' corresponding to container PIDs. - cgroup memory.max for the container path reveals a low effective limit (e.g., ~1Gi) despite pod not specifying a memory limit. - kubelet logs show 'cgroup driver: systemd' and a warning about failing to apply memory limit to system.slice when node memory swap accounting disabled. - Pod's container runtime inspect shows memory limit set to a lower value by CRI; inspecting the pod YAML shows only requests: 512Mi, no limits. - Prometheus Node exporter metrics show memory.available high and no node OOM conditions preceding the container kills.

Already attempted

Added explicit memory.limits in pod spec to 1Gi; behavior persisted for some pods until limit enforced and then OOM no longer occurred for properly sized containers. - Evicted other low-importance pods to rule out contention. - Rebooted node and recreated pods; problem reappeared on same node. - Compared cgroup driver and kubelet --cgroup-driver settings across nodes; same configuration. - Disabled swap on node (swap was already off).

Question

Why are containers being OOM-killed even though the host reports plenty of free RAM? What node-level or kubelet cgroup configuration should be checked or corrected so that Kubernetes respects pod memory requests/limits and prevents host OOM-kills?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Systemd slice units (e.g., docker.service or system.slice) have MemoryMax set low, causing child container cgroups to inherit tight limits 85%

CRI (containerd) or a node bootstrap script sets default runtime memory limit for containers when no limit specified 75%

Kernel memory accounting (CONFIG_MEMCG) or swap accounting disabled causing kubelet to fall back to systemd-enforced limits 70%

Summary

The kernel OOM killer is targeting container processes because the container's cgroup memory limit is lower than expected. When pods omit memory limits, the container runtime may still apply default cgroup limits derived from node or kubelet configuration (e.g., kubelet's --cgroup-root or a systemd slice with a small Limit). A common root cause is misconfiguration between kubelet and container runtime cgroup drivers (systemd vs cgroupfs), or a systemd slice with low memory.limit_in_bytes applied to the container's parent slice. Another possibility is node-level memory enforcement via a memory balloon or VM-level limits that reduce available memory for cgroups. The fact explicit limits change behavior indicates cgroup controls are effective and current defaults are too low.

Suggested Fixes

Diagnostic step 1

Confirm cgroup drivers: check kubelet --cgroup-driver and containerd config (sudo cat /etc/containerd/config.toml) and ensure both use 'systemd' or 'cgroupfs' consistently; avoid changing in production without testing

Diagnostic step 2

Inspect effective cgroup limits for a live container: identify container PID, then check memory.max and memory.usage_in_bytes under /sys/fs/cgroup/<slice>/... and compare to expected values

Diagnostic step 3

Check systemd slice configs for MemoryMax: systemctl show --property=MemoryAccounting yes and inspect /etc/systemd/system/*.slice for MemoryMax entries; if present and inappropriate, test removing or raising them and restart services (reversible by restoring original files)