1
Diagnostic step 1
Enable and collect detailed CNI plugin logs (Calico) and kubelet logs around pod start and deletion events to correlate with DNS timeouts. These logs are read-only to collect.
FULL AI ANALYSIS
GKE cluster DNS fails inside pods intermittently after node auto-scaler removes older node pool
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
CoreDNS timeouts and node-local-dns stale entries suggest an interaction between node churn and DNS path (either node-local cache or CNI networking setup) during or immediately after autoscaler operations. Observed CNI errors on new nodes and pod-level UDP DNS requests not receiving timely replies point to a transient networking path break (iptables, routes, or CNI programming delay) rather than CoreDNS process crash. Rolling pods resolve on restart, consistent with reprogramming of network paths.
Possible causes
Timing race where CNI plugin hasn't fully programmed pod networking and iptables/nft rules before pod attempts outbound DNS, causing DNS UDP packets to be dropped.
85%Cluster autoscaler drain timing combined with NetworkPolicy enforcement briefly blocks DNS egress for pods being migrated.
60%CoreDNS upstream servers (e.g., Cloud DNS) temporarily rate-limit or drop UDP queries during scale events, causing retransmits to fail.
55%Node-local-dns caching layer references IPs of deleted CoreDNS endpoints or the local cache isn't invalidated promptly after node removal.
35%Suggested fixes
2
Diagnostic step 2
Temporarily increase cluster-autoscaler 'scaleDownUnneededTime' to reduce node churn and observe if DNS incidents drop; this is reversible by restoring the setting later.
3
Diagnostic step 3
Add a readiness probe that ensures DNS (e.g., dig @127.0.0.1 service.cluster.local) succeeds before marking pods Ready, so application pods delay outbound DNS until networking is stable.
4
Diagnostic step 4
Instrument CoreDNS with Prometheus metrics for UDP errors and increase replica count in a controlled way to see if higher capacity reduces timeouts.
5
Diagnostic step 5
If node-local-dns is in use, enable verbose logging for it or temporarily route pods to use CoreDNS directly to test whether node-local cache is the cause; collect pdump/tcpdump during replication of the issue and, if confirmed, consider filing a GKE support case with packet captures and control-plane logs for further investigation.