1
Diagnostic step 1
On an affected node, compare kubelet cgroup driver to Docker: run 'ps -p 1 -o comm=' to confirm init and check 'docker info | grep -i cgroup' and kubelet flags (--cgroup-driver). Do not change config yet; just record outputs.
FULL AI ANALYSIS
Kubernetes pods OOMKilled while node shows 80GB free memory on Ubuntu 20.04 (kubelet+Docker cgroup mismatch suspected)
Generated Oct 14, 2026, 9:39:02 AM
Analysis summary
Symptoms point to OOM kills occurring inside container cgroups while host-wide memory remains available. Common causes are (a) mismatch between kubelet and container runtime cgroup drivers or cgroup v1/v2 differences so memory limits are not consistently applied, (b) kernel memory accounting or swap settings altering when the kernel OOM killer runs, or (c) mis-declared container limits/typos in manifests causing some containers to have no enforced limits and then hit process-level limits. The presence of OOM messages in dmesg referencing container PIDs plus missing/empty cgroup memory files for some containers increases suspicion on runtime/kubelet cgroup setup after recent system updates.
Possible causes
Kubelet and Docker cgroup driver mismatch (systemd vs cgroupfs) or partial migration to cgroup v2 causing memory limits not uniformly enforced
80%Kernel memory accounting disabled or swap settings allow host overcommit, so kernel OOM killer selects container processes despite free host memory
55%Deployment manifests have mis-specified memory limits for some replicas (typos, missing units, or templating bug) so those containers hit their per-container limits
45%Application-level memory leak that eventually exceeds the container's configured limit (per-container enforcement still working for some pods)
35%Suggested fixes
2
Diagnostic step 2
Inspect cgroup tree for an OOMKilled container: find the container ID from 'kubectl describe pod', then check /sys/fs/cgroup/memory/docker/<id>/memory. View memory.limit_in_bytes and memory.max_usage_in_bytes. If those files are missing or inconsistent between containers, collect findings.
3
Diagnostic step 3
Check kernel OOM messages: run 'dmesg --ctime | grep -i oom' and cross-reference killed PIDs with container PIDs (pod UID shown in kubectl describe). Save logs for analysis.
4
Diagnostic step 4
Confirm whether the system moved to cgroup v2: test presence of /sys/fs/cgroup/cgroup.controllers and check 'mount | grep cgroup'. If cgroup version changed recently, plan for coordinated kubelet/runtime config change; do not change both on production nodes without maintenance window.
5
Diagnostic step 5
Validate resource manifests: use 'kubectl get deploy -o yaml' to confirm limits are present and correctly unit-suffixed (e.g., '2Gi', not '2000M' if templating may alter units). Use 'kubectl describe pod' on multiple replicas to compare container limits at runtime before editing manifests permanently (reversible).","If evidence points to kernel/runtime-level accounting issues (cgroup mismatch or cgroup v2 transition), escalate to cluster infrastructure team to schedule a rolling node maintenance: align kubelet cgroup-driver with Docker or migrate container runtime to a consistent cgroup v2-aware configuration. Changes at this level should be tested in staging and rolled with node drains; consider opening a support ticket with your runtime vendor if unsure.