GKE cluster DNS fails inside pods intermittently after node auto-scaler removes older node pool

Environment

Google Kubernetes Engine (GKE) Autopilot cluster v1.25.7-gke.1200 with two node pools: pool-a (n1-standard-4) and pool-b (n2-standard-2). Cluster uses CoreDNS (coredns/coredns:1.8.4) as DNS add-on, Cloud NAT for outbound internet, Calico CNI enabled. HorizontalPodAutoscaler and cluster autoscaler are active; pods are scheduled across both pools.

Symptoms

After scale-down events where the autoscaler drains and deletes older nodes in pool-b, some newly scheduled pods intermittently fail DNS resolution for external hostnames (e.g., api.example.com). Symptoms last 1–3 minutes for affected pods; a restart of the pod usually restores DNS resolution. Internal Kubernetes DNS lookups (svc.cluster.local) remain unaffected.

Observations

CoreDNS pod logs show occasional 'read udp ... i/o timeout' and 'context deadline exceeded' errors. kube-proxy logs on deleted nodes show graceful drain was requested; however, iptables rules and node-local DNS caches (node-local-dns enabled) sometimes show stale entries referencing IP addresses of deleted nodes. NetworkPolicy enforcement logs show endpoints moved during drain. Not all pods on the same node are affected; primarily newly scheduled pods on nodes created shortly after a scale-down event show DNS issues. Kubelet logs on a recently created node show transient CNI plugin errors during pod start.

Already attempted

Rolled CoreDNS to a higher replica count; upgraded CoreDNS image to a later patch (tested in staging); disabled node-local-dns temporarily (no effect). Increased CoreDNS upstream timeout and cache TTL. Tuned cluster-autoscaler scale-down delay to longer window to reduce churn. Recreated failed pods manually and they regained DNS. Captured tcpdump on node interfaces during failures and saw DNS UDP requests from pods that did not get replied to within the node's network stack timeframe occasionally.

Question

What targeted debugging steps can confirm whether DNS timeouts are caused by CNI race conditions during node replacement, leftover iptables/nft rules, node-local caching inconsistencies, or CoreDNS upstream saturation? Which reversible cluster-level configuration changes are safest to test in production to reduce or eliminate these transient DNS failures?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Timing race where CNI plugin hasn't fully programmed pod networking and iptables/nft rules before pod attempts outbound DNS, causing DNS UDP packets to be dropped. 85%

Cluster autoscaler drain timing combined with NetworkPolicy enforcement briefly blocks DNS egress for pods being migrated. 60%

CoreDNS upstream servers (e.g., Cloud DNS) temporarily rate-limit or drop UDP queries during scale events, causing retransmits to fail. 55%

Summary

CoreDNS timeouts and node-local-dns stale entries suggest an interaction between node churn and DNS path (either node-local cache or CNI networking setup) during or immediately after autoscaler operations. Observed CNI errors on new nodes and pod-level UDP DNS requests not receiving timely replies point to a transient networking path break (iptables, routes, or CNI programming delay) rather than CoreDNS process crash. Rolling pods resolve on restart, consistent with reprogramming of network paths.

Suggested Fixes

Diagnostic step 1

Enable and collect detailed CNI plugin logs (Calico) and kubelet logs around pod start and deletion events to correlate with DNS timeouts. These logs are read-only to collect.

Diagnostic step 2

Temporarily increase cluster-autoscaler 'scaleDownUnneededTime' to reduce node churn and observe if DNS incidents drop; this is reversible by restoring the setting later.

Diagnostic step 3

Add a readiness probe that ensures DNS (e.g., dig @127.0.0.1 service.cluster.local) succeeds before marking pods Ready, so application pods delay outbound DNS until networking is stable.