1
Diagnostic step 1
Confirm the client uses HTTP/2 and correct host header: run openssl s_client -connect LB:PORT -servername service.example.com and observe ALPN protocols, or use grpcurl with -authority and -insecure flags to test.
FULL AI ANALYSIS
Kubernetes Ingress returns 404 only for external gRPC clients behind cloud load balancer
Generated Oct 3, 2026, 12:35:03 AM
Analysis summary
404s for external gRPC but successful internal calls indicate that traffic path or request metadata is modified by the LB or ingress before reaching the gRPC backend. Common culprits: L7 load balancer terminating TLS and not preserving the HTTP/2 framing or host header required for ingress routing, ingress controller lacking correct grpc backend-protocol annotation or not receiving correct host:port, or client using HTTP/1.1 (via TLS termination) which ingress then treats as a standard HTTP request and routes to default backend. The fact TCP passthrough reduced to TLS issues suggests LB termination is a key variable.
Possible causes
Ingress hostname or host header mismatch: LB rewrites Host header to IP or different hostname, causing ingress to select default backend returning 404.
90%Load balancer terminates TLS and converts HTTP/2 gRPC to HTTP/1.1, causing ingress to treat requests as HTTP and not match gRPC-specific routing rules.
40%Health-check path and LB backend routing misconfiguration sends traffic to wrong node/port or to ingress default backend rather than ingress controller.
30%Ingress controller missing or ignoring backend-protocol: GRPC annotation (misconfigured ConfigMap or older ingress-controller incompatible with annotation).
25%Suggested fixes
2
Diagnostic step 2
Temporarily configure the LB to use TCP passthrough to a single ingress node and use TLS passthrough (not terminate) so TLS and HTTP/2 are preserved; test gRPC after ensuring server cert is valid. If passthrough succeeds, LB termination is the issue and LB settings must preserve HTTP/2 and host header.
3
Diagnostic step 3
Add a distinct ingress rule that matches the LB’s observed Host header (or set ingress to accept both hostnames) to rule out hostname mismatch; do not delete production rules — apply in a test namespace.
4
Diagnostic step 4
Enable nginx-ingress debug logging and capture the full request headers on ingress to see Host, :scheme, and HTTP version. Check ingress-controller ConfigMap for use-http2 and proxy protocol settings.
5
Diagnostic step 5
If LB cannot preserve HTTP/2 semantics, consider switching to a TCP/L4 load balancer or using a gRPC-aware gateway (Envoy/Contour) that explicitly supports gRPC proxying. If debugging shows LB rewrites and vendor docs are unclear, escalate to cloud provider support with packet traces (no sensitive payload) and ingress logs. Avoid changing TLS/keys in production without backups.