Kubernetes admission webhook causes API server 404s for gRPC services only when behind AWS NLB with proxy-protocol enabled

Environment

EKS cluster: Kubernetes 1.23 control plane, AWS NLB in front of nginx-ingress-controller Service (type=LoadBalancer) with proxy-protocol v2 enabled; several apps expose gRPC services via HTTP/2 through the ingress.

Symptoms

External gRPC clients intermittently receive 404 from the ingress for long-running gRPC streams; short unary calls succeed. Logs show the ingress pod returning 404 for requests that should route to backends. Direct cluster-internal calls to the Service succeed.

Observations

nginx-ingress-controller logs show HTTP/2 connection resets and 'client sent HTTP/1.1 request without upgrade' style messages preceding 404s. The AWS NLB was recently configured to enable proxy-protocol to preserve client IPs, and the Service annotation on the ingress has service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'. Captured TCP payloads at ingress node show proxy-protocol v2 header present on some connections while absent on others (connections from NLB cross AZs). The admission webhook for validating TLS certificates is configured cluster-wide and returns 200/denied logs show no rejections. No 502/504 errors, only 404 on specific long-lived gRPC streams.

Already attempted

Removed and re-added the proxy-protocol annotation on the Service — temporarily restored some traffic but 404s returned. Disabled the admission webhook (put it in maintenance mode) — no change. Scaling nginx-ingress to two replicas changes which connections hit which pod; 404s follow the pattern of connections arriving with or without proxy-protocol headers.

Question

Why are gRPC streams seeing 404s only when proxy-protocol is toggled on the NLB, and how can I configure the load balancer and nginx-ingress so HTTP/2 gRPC traffic stays intact across AZs without turning off client IP preservation?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

nginx-ingress is not configured to accept proxy-protocol v2 on the port used for HTTP/2/gRPC, so when header appears, nginx fails to parse the HTTP/2 stream correctly. 95%

NLB is inconsistently sending proxy-protocol headers (some targets/AZs configured differently) causing nginx to misinterpret HTTP/2 preface as header or vice versa. 85%

A mix of direct node-targeted connections and NLB-targeted connections leads to some traffic arriving without proxy-protocol header. 40%

Summary

The symptoms indicate that some connections arrive at nginx with or without the proxy-protocol header depending on how the NLB forwards traffic across AZs or during failover; if nginx is configured to expect proxy-protocol but receives plain TCP, or vice versa, the parsing of the initial bytes (proxy header vs HTTP/2 preface) misinterprets the stream leading to protocol framing errors and 404s. gRPC over HTTP/2 is sensitive to the connection preface. The intermittent nature corresponds to mixed presence of proxy-protocol (NLB sometimes not sending header for certain connections or client connections bypassing NLB behavior). The admission webhook likely unrelated. The correct fix is to ensure consistent proxy-protocol handling across all NLB listeners and that the ingress controller is configured to accept proxy-protocol v2 for all frontends.

Suggested Fixes

Diagnostic step 1

Verify NLB listener and target group configurations across all AZs: ensure proxy-protocol is enabled uniformly and the target groups are the same for each listener. Check AWS console and CLI (aws elbv2 describe-listeners, describe-target-groups).

Diagnostic step 2

Enable proxy-protocol v2 support in nginx-ingress for the Service ports handling gRPC. Add the annotation service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' and configure the ingress controller with --enable-proxy-protocol and ensure it listens for v2 headers on the correct ports.

Diagnostic step 3

Confirm the ingress nginx config uses 'listen 443 proxy_protocol' and set real_ip_header proxy_protocol so nginx parses the proxy header and preserves HTTP/2 preface. Reload nginx-ingress after changes.