CoreDNS resolution breaks inside pods after kube-proxy switch to IPVS on Debian worker nodes

Environment

On-premises k8s: control plane on Ubuntu 20.04, worker nodes running Debian 11, kube-proxy switched from iptables to IPVS mode via kubeadm; CoreDNS v1.8.0 in kube-system; Calico CNI in use (policy only), nodes use systemd-resolved locally.

Symptoms

Within several pods, DNS lookups for cluster services and external domains intermittently time out; DNS queries succeed from host (dig @127.0.0.53) but fail from affected pods. Some pods can reach 10.96.0.10 (ClusterIP) but queries to CoreDNS take >5s or get SERVFAIL.

Observations

kube-proxy logs show ipvs controller errors on two Debian workers: 'failed to sync ipvs service' and 'netlink: operation not supported'. ipvsadm on those nodes lists incomplete services. Kernel modules ip_vs and nf_conntrack appear loaded on Ubuntu nodes but missing on problematic Debian nodes. sysctl net.bridge.bridge-nf-call-iptables is 0 on affected nodes.

Already attempted

Restarted kube-proxy daemonset pods — transient improvement for a minute then failures resumed. Re-applied Calico manifests. Tried restarting kubelet and resolvedd on host. Gathered strace of kube-proxy showing failures when creating IPVS rules. Did not reboot nodes yet.

Question

Which node-level kernel features and IPVS prerequisites should be checked and how can I roll back to iptables safely if IPVS cannot be stabilized, without disrupting running workloads?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

Missing or unloaded kernel IPVS modules (ip_vs, ip_vs_rr, nf_conntrack) on Debian workers. 95%

Kernel compiled without required IPVS support or running an older kernel lacking necessary features. 80%

kube-proxy IPVS mode incompatible with underlying distributions' netfilter setup or a custom kernel/packaging difference on Debian nodes. 75%

Summary

kube-proxy in IPVS mode relies on kernel IPVS modules and proper netfilter/sysctl settings; errors and missing modules on Debian workers indicate the nodes lack required kernel support or configuration. Because host-level DNS works while in-pod resolution fails, the issue is at the kube-proxy/IPVS service level creating virtual IPs for ClusterIP services. Reverting to iptables is a safe mitigation when IPVS prerequisites cannot be satisfied quickly.

Suggested Fixes

Diagnostic step 1

On the affected Debian nodes, check kernel modules: lsmod | egrep 'ip_vs|nf_conntrack' and load if missing (modprobe ip_vs; modprobe ip_vs_rr; modprobe nf_conntrack_netlink).

Diagnostic step 2

Verify kernel supports IPVS: review dmesg for ipvs errors and check uname -a; if custom kernel lacking IPVS, plan a kernel upgrade or fallback.

Diagnostic step 3

Ensure sysctls are set: sysctl net.bridge.bridge-nf-call-iptables=1 and net.ipv4.ip_forward=1. Apply temporarily and test.