Kubernetes: ConfigMap updates not reflected in pods using projected volume unless pod restarted (changes appear after 2–3 minutes only sometimes)

Environment

On-prem K8s cluster v1.25 with kubelet on Ubuntu 22.04 nodes, pods mount ConfigMap via projected volume (volumeMount to /app/config). Pod runs a Java application under OpenJDK 17 that reads configuration files on startup. ConfigMap updated via 'kubectl apply', but pods sometimes do not see changes until restarted; other times change appears after ~3 minutes.

Symptoms

Operators update ConfigMap with new configuration; expected behavior is k8s kubelet to update the projected volume contents within seconds. In observed cases, files in /app/config inside pod remain unchanged. Running 'kubectl describe pod' shows configMap generation annotation updated, but container file contents stale. Restarting pod immediately reflects new config. On other pods on different nodes the update propagates quickly.

Observations

kubelet logs on affected node show 'ConfigMap not mounted because path already exists' and occasional errors 'failed to remove old files during atomic update' while kubelet attempts to sync. Node runs containerd 1.6 and uses overlayfs for containerd storage. In the pod, inotify does not reliably report changes (tested with inotifywait). The projected volume implementation updates via writing to a tmp dir and renaming; overlayfs/atomic rename semantics combined with container runtime can yield race conditions. There are no network issues between API server and kubelet; the problem correlates with high disk I/O and nodes with heavy overlayfs activity.

Already attempted

Tested with a fresh pod on same node (new pod saw config immediately). For affected pods, issuing 'kubectl rollout restart' or deleting pod fixed behavior. Upgrading kubelet was not possible in production; tried deleting and recreating the ConfigMap (same result). Checked file permissions and SELinux/AppArmor not blocking. Reproduced the issue on a staging node loaded with container image pulls and overlayfs churn — reproduction rate higher under disk I/O.

Question

What are the reliable, low-risk mitigations to ensure ConfigMap projected volumes are atomically and rapidly updated inside pods (especially for apps that must pick up changes without restart), and how can I diagnose whether overlayfs atomic replace semantics on the node's container runtime are the root cause?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Overlayfs semantics on nodes with heavy I/O prevent atomic rename from making updated files visible inside the container immediately. 90%

Kubelet encountered transient errors cleaning up temporary files during ConfigMap update leading to inconsistent state on the node. 75%

Application relies on inotify which may not see events if kubelet replaced files with rename semantics that bypass inotify on the container side. 70%

Summary

Kubernetes updates projected ConfigMap volumes using a write-to-temp and rename pattern. On overlayfs-backed container rootfs, atomic rename semantics can interact poorly with existing mountpoints and overlay layer caching, causing kubelet to fail to make the new file visible to the container's mount namespace until remediation (pod restart) or until overlayfs refresh occurs. High disk I/O increases likelihood of these race conditions. Diagnosis points to node-level overlayfs behavior and kubelet sync failure messages. Workarounds include using non-projected mechanisms or ensuring application watches for config changes robustly.

Suggested Fixes

Diagnostic step 1

Avoid relying on in-place ConfigMap updates for hot-reload in production-critical services: implement an explicit sidecar that watches the API (watch ConfigMaps) and writes configuration in-place inside container filesystem in a way the application expects, or use a shared volume that the application polls. These are reversible code/deployment changes.

Diagnostic step 2

For immediate mitigation, perform a rolling restart of affected deployments after a ConfigMap change ('kubectl rollout restart deployment/...') until the underlying bug is addressed.

Diagnostic step 3

Collect kubelet logs from the affected node around ConfigMap update time (journalctl -u kubelet) and gather containerd and dmesg output showing overlayfs messages; reproduce on a staging node with controlled overlayfs churn to confirm root cause, then open a bug with Kubernetes with the logs if reproducible.