FULL AI ANALYSIS

Kubernetes admission webhook causes API server 404s for gRPC services only when behind AWS NLB with proxy-protocol enabled

Generated Oct 3, 2026, 12:35:03 AM

Analysis summary

The symptoms indicate that some connections arrive at nginx with or without the proxy-protocol header depending on how the NLB forwards traffic across AZs or during failover; if nginx is configured to expect proxy-protocol but receives plain TCP, or vice versa, the parsing of the initial bytes (proxy header vs HTTP/2 preface) misinterprets the stream leading to protocol framing errors and 404s. gRPC over HTTP/2 is sensitive to the connection preface. The intermittent nature corresponds to mixed presence of proxy-protocol (NLB sometimes not sending header for certain connections or client connections bypassing NLB behavior). The admission webhook likely unrelated. The correct fix is to ensure consistent proxy-protocol handling across all NLB listeners and that the ingress controller is configured to accept proxy-protocol v2 for all frontends.

Possible causes

nginx-ingress is not configured to accept proxy-protocol v2 on the port used for HTTP/2/gRPC, so when header appears, nginx fails to parse the HTTP/2 stream correctly.

95%

NLB is inconsistently sending proxy-protocol headers (some targets/AZs configured differently) causing nginx to misinterpret HTTP/2 preface as header or vice versa.

85%

A mix of direct node-targeted connections and NLB-targeted connections leads to some traffic arriving without proxy-protocol header.

40%

TLS termination mismatch or premature connection close upstream results in 404s appearing for long-lived streams rather than proper 502/495 errors.

20%

Suggested fixes

Diagnostic step 1

Verify NLB listener and target group configurations across all AZs: ensure proxy-protocol is enabled uniformly and the target groups are the same for each listener. Check AWS console and CLI (aws elbv2 describe-listeners, describe-target-groups).

Diagnostic step 2

Enable proxy-protocol v2 support in nginx-ingress for the Service ports handling gRPC. Add the annotation service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' and configure the ingress controller with --enable-proxy-protocol and ensure it listens for v2 headers on the correct ports.

Diagnostic step 3

Confirm the ingress nginx config uses 'listen 443 proxy_protocol' and set real_ip_header proxy_protocol so nginx parses the proxy header and preserves HTTP/2 preface. Reload nginx-ingress after changes.

Diagnostic step 4

Capture packets at the node where ingress pod runs to confirm presence and format (v1 vs v2) of proxy headers for failing connections. Use tcpdump and examine initial bytes to correlate failures.

Diagnostic step 5

If client IP preservation is not essential, test disabling proxy-protocol on NLB to verify the issue disappears; use that as diagnostic. If proxy-protocol must remain, consider using an ALB (which supports HTTP/2 natively to targets) or terminate TLS at the load balancer and forward HTTP/2 properly to backends.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads