Kubernetes pods OOMKilled while node shows 80GB free memory on Ubuntu 20.04 (kubelet+Docker cgroup mismatch suspected)

Environment

On-prem Kubernetes 1.23 control plane, 5 worker nodes (Ubuntu 20.04, Docker 20.10.12, kubelet configured with cgroupDriver=systemd). Workers: 128GB RAM, 32 vCPU. Applications deployed as Deployments with requests/limits set (example: request=512Mi, limit=2Gi). No node swap configured in cloud-init.

Symptoms

Application Pods abruptly terminate with status OOMKilled during short CPU/memory bursts. Restarts clustered to particular Deployments during load spikes; problem appears after a recent OS package update and Docker restart. End users report 502s from the service for ~10–60s until pods restart.

Observations

kubectl describe pod shows ContainerStatus: Terminated, Reason=OOMKilled, ExitCode=137. top/htop on node shows >80GB free memory. free -m and vmstat show low overall memory pressure. dmesg contains OOM kill messages referencing PIDs belonging to container processes. kubectl top node reports substantial usage but not exhausted. /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes exists and matches expected limit for some containers; for others the cgroup path is missing or not populated.

Already attempted

Restarted Docker and kubelet on affected nodes (briefly mitigated for 1–2 hours). Scaled the Deployments up/out to distribute load (reduced frequency but not eliminated OOMs). Added temporary higher memory limits in Deployment yaml and reapplied (containers still OOMKilled after a short period). Ran kubelet --version and docker info to confirm versions; reviewed recent apt upgrade list (kernel and systemd packages updated).

Question

What is the most likely reason Kubernetes containers are being OOMKilled when the host shows plenty of free RAM, and what safe diagnostics and reversible configuration checks should I run to identify whether this is a cgroup accounting/enforcement problem versus an application memory leak?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

Kubelet and Docker cgroup driver mismatch (systemd vs cgroupfs) or partial migration to cgroup v2 causing memory limits not uniformly enforced 80%

Kernel memory accounting disabled or swap settings allow host overcommit, so kernel OOM killer selects container processes despite free host memory 55%

Deployment manifests have mis-specified memory limits for some replicas (typos, missing units, or templating bug) so those containers hit their per-container limits 45%

Summary

Symptoms point to OOM kills occurring inside container cgroups while host-wide memory remains available. Common causes are (a) mismatch between kubelet and container runtime cgroup drivers or cgroup v1/v2 differences so memory limits are not consistently applied, (b) kernel memory accounting or swap settings altering when the kernel OOM killer runs, or (c) mis-declared container limits/typos in manifests causing some containers to have no enforced limits and then hit process-level limits. The presence of OOM messages in dmesg referencing container PIDs plus missing/empty cgroup memory files for some containers increases suspicion on runtime/kubelet cgroup setup after recent system updates.

Suggested Fixes

Diagnostic step 1

On an affected node, compare kubelet cgroup driver to Docker: run 'ps -p 1 -o comm=' to confirm init and check 'docker info | grep -i cgroup' and kubelet flags (--cgroup-driver). Do not change config yet; just record outputs.

Diagnostic step 2

Inspect cgroup tree for an OOMKilled container: find the container ID from 'kubectl describe pod', then check /sys/fs/cgroup/memory/docker/<id>/memory. View memory.limit_in_bytes and memory.max_usage_in_bytes. If those files are missing or inconsistent between containers, collect findings.

Diagnostic step 3

Check kernel OOM messages: run 'dmesg --ctime | grep -i oom' and cross-reference killed PIDs with container PIDs (pod UID shown in kubectl describe). Save logs for analysis.