Kubernetes pods CrashLoopBackOff only on upgraded nodes with hugepages enabled — short lived OOM at container startup

Environment

GKE-like cluster: two node pools — old pool (Ubuntu 18.04, no hugepages) where workload stable, upgraded node pool (Ubuntu 20.04, kernel 5.4) configured with hugepages-1G reserved (systemd hugepages mount) via kubelet and cloud-init; pods are general-purpose Java and Python services without explicit hugepage requests.

Symptoms

After node pool upgrade, certain pods scheduled to the new nodes enter CrashLoopBackOff within seconds of launch. Logs in the failed pod show OOMKilled or 'malloc: cannot allocate memory' during JVM bootstrap. Same pod runs fine on old nodes. Restarting node moves pod to same node and repeats failure. Node metrics show significant hugepages reserved and kubelet reports Allocatable memory decreased.

Observations

Nodes with failures show hugepages configured in /sys/kernel/mm/hugepages and kubelet args include --hugepages-1G=1Gi. 2) kubectl describe pod shows OOMKilled but container memory limits not exceeded. 3) dmesg on node shows 'out of memory' events and kernel allocating hugepages at boot reduced the memory available to the general allocator. 4) Pods do not request hugepages via resource.requests or limits. 5) Eviction thresholds on kubelet are unchanged, but Allocatable memory reported to the scheduler is lower on upgraded nodes.

Already attempted

Scaled down and then back up affected deployments; moved deployments by adding nodeSelector to schedule to older pool (works). Removed hugepages configuration temporarily on one test node and pods began running normally. Verified no cgroup memory limits set at pod level that would explain immediate OOM.

Question

Why are ordinary pods OOM-killed on nodes that have hugepages configured even though they don't request hugepages? How should hugepages be enabled safely without breaking existing workloads?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 3h ago• Trouble ID

Most likely causes

Host-level hugepages reservation reduces available general memory; scheduler may still schedule pods whose runtime memory needs cannot be satisfied leading to immediate OOM/kills 90%

Kubelet allocatable reporting misconfiguration — kubelet not informing scheduler accurately about reduced memory due to hugepages, leading to overscheduling 50%

JVM/Python startup large allocations require contiguous memory and fail due to fragmentation after hugepages allocation, even though available memory metric looks sufficient 40%

Summary

Enabling hugepages at the host level reduces the general-purpose RAM available to the kernel and userland allocators — hugepages are reserved and not available for normal allocations unless containers request them. The kubelet reports a reduced Allocatable memory to the scheduler, but if existing DaemonSets or system components consume more memory or kubelet accounting differs, pods can still be scheduled despite insufficient usable memory. Additionally, the kernel's handling of hugepages allocation and fragmentation at boot can cause lower usable memory for large allocations (JVM), resulting in OOM at startup. The test that removing hugepages made pods healthy strongly points to host-level hugepage reservation as the root cause rather than container configuration.

Suggested Fixes

Diagnostic step 1

Do not enable hugepages cluster-wide on nodes that run unmodified workloads. Instead create a dedicated node pool (taint the nodes and use tolerations/nodeSelectors) for workloads that explicitly request hugepages so existing pods are not scheduled there.

Diagnostic step 2

On affected nodes, check kubectl describe node and compare 'Allocatable' memory before and after hugepages are enabled. Verify kubelet flags and ensure Kubelet advertises correct resources (including hugepages) to the scheduler.

Diagnostic step 3

Use node taints to prevent general workloads from landing on hugepage nodes, and require workloads that need hugepages to declare hugepages in resource.requests/limits following Kubernetes hugepages docs.