Kubernetes Ingress returns 404 only for external gRPC clients behind cloud load balancer

Environment

K8s 1.25 cluster on managed control plane, nginx-ingress-controller v1.6 deployed as DaemonSet, services expose containerPort 50051 gRPC; internal cluster clients can call gRPC services successfully (kubectl port-forward and in-cluster pods), external calls from cloud L7 TCP/HTTP(S) load balancer return 404 from ingress.

Symptoms

External gRPC clients attempting to connect over TLS to LB IP:PORT receive HTTP/1.1 404 responses or gRPC status UNIMPLEMENTED; telemetry shows client establishing TLS then immediate 404. Internal tests (curl grpc-health-check via port-forward) succeed.

Observations

nginx-ingress logs show requests hitting other hostnames or default backend for same path, with '404 host not found' entries. The LB is configured with a target of nodePort 30080 and health checks hitting /healthz over HTTP; LB is terminating TLS and forwarding plaintext to nodes. Ingress resource uses 'kubectl describe' and shows hostname 'service.example.com' and annotation ingress.kubernetes.io/backend-protocol: 'GRPC'. ClusterInternal DNS resolves correctly.

Already attempted

Verified service and endpoints; re-deployed ingress with nginx.ingress.kubernetes.io/backend-protocol: 'GRPC' annotation. Switched LB to TCP passthrough mode (no TLS termination) — gRPC connects but has TLS handshake issues since client expects server cert. Mapped LB to a single node and port via nodePort — connection still yields 404. Increased ingress log level and checked ingress-controller configmap for server-snippet — no custom rewrites. Recreated ingress on a test namespace with identical manifests — same 404.

Question

What sequence of reversible checks will identify whether the 404 is caused by the external load balancer (TLS termination/host header rewrite), ingress controller configuration (protocol annotations), or Ingress resource hostname mismatch, and how can I route external gRPC traffic reliably through the current ingress stack?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

Ingress hostname or host header mismatch: LB rewrites Host header to IP or different hostname, causing ingress to select default backend returning 404. 90%

Load balancer terminates TLS and converts HTTP/2 gRPC to HTTP/1.1, causing ingress to treat requests as HTTP and not match gRPC-specific routing rules. 40%

Health-check path and LB backend routing misconfiguration sends traffic to wrong node/port or to ingress default backend rather than ingress controller. 30%

Summary

404s for external gRPC but successful internal calls indicate that traffic path or request metadata is modified by the LB or ingress before reaching the gRPC backend. Common culprits: L7 load balancer terminating TLS and not preserving the HTTP/2 framing or host header required for ingress routing, ingress controller lacking correct grpc backend-protocol annotation or not receiving correct host:port, or client using HTTP/1.1 (via TLS termination) which ingress then treats as a standard HTTP request and routes to default backend. The fact TCP passthrough reduced to TLS issues suggests LB termination is a key variable.

Suggested Fixes

Diagnostic step 1

Confirm the client uses HTTP/2 and correct host header: run openssl s_client -connect LB:PORT -servername service.example.com and observe ALPN protocols, or use grpcurl with -authority and -insecure flags to test.

Diagnostic step 2

Temporarily configure the LB to use TCP passthrough to a single ingress node and use TLS passthrough (not terminate) so TLS and HTTP/2 are preserved; test gRPC after ensuring server cert is valid. If passthrough succeeds, LB termination is the issue and LB settings must preserve HTTP/2 and host header.

Diagnostic step 3

Add a distinct ingress rule that matches the LB’s observed Host header (or set ingress to accept both hostnames) to rule out hostname mismatch; do not delete production rules — apply in a test namespace.