ALB returns 502 for gRPC over HTTP/2 after enabling TLS policy to TLS1.3-only

Environment

AWS Application Load Balancer (ALB) frontend configured for TLS termination, target group is gRPC server (HTTP/2) on Fargate tasks. Previously used TLS policy supporting TLS1.2 & TLS1.3. After switching ALB listener to a custom policy allowing only TLS1.3, clients see intermittent 502 responses for gRPC calls while health checks to /healthz still pass.

Symptoms

gRPC clients report UNAVAILABLE with ALB returning 502 Bad Gateway for some streams; other gRPC calls succeed intermittently. ALB access logs show target response codes 502 with 'cause' hinting at HTTP/2 framing errors. Backend tasks' application logs show no corresponding request entries for failed requests. TLS handshakes from ALB to backend are not used because ALB terminates TLS and forwards to backend over HTTP/2 plain text.

Observations

Rolling back ALB TLS policy to include TLS1.2 resolves the 502 errors immediately. No change to backend task configuration. ALB metrics show increased TLS negotiation timeouts and some TLS protocol errors shortly after policy change. Client connections use TLS1.3 to the ALB. ALB target group protocol is set to HTTP/2 with 'protocol version' ALPN negotiation expected between client and ALB, and ALB performs HTTP/2 to backend. No custom proxies in between ALB and targets.

Already attempted

Reverted ALB TLS policy to previous TLS1.2+TLS1.3 policy (issue stopped). Increased backend connection timeouts. Rotated ALB certificates. Checked application logs for malformed requests — none found. Verified ALB health checks continued to return 200 on backend.

Question

Why would enabling a TLS1.3-only policy on ALB cause intermittent 502 for gRPC traffic even though ALB is terminating TLS? What diagnostics should I collect and what configuration changes are safe to try while minimizing disruption to production traffic?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

ALB internal bug or limitation handling HTTP/2/gRPC when frontend uses TLS1.3-only policy, causing misinterpretation of ALPN or streams. 85%

Cipher suite selection under TLS1.3-only policy results in unexpected record framing from ALB's TLS termination path, exposing an ALB implementation edge-case. 60%

Intermittent TLS handshakes failing due to buggy client-side TLS version implementations leading to half-open connections presented to ALB. 45%

Summary

ALB acts as TLS terminator and speaks HTTP/2 to the backend. Switching to TLS1.3-only changes the ALPN and handshake characteristics between clients and ALB. Possible causes include ALB internal handling bugs for HTTP/2 over TLS1.3, ALB cipher suite mismatches causing early connection failures that show as 502, or client ALPN mismatch leading to ALB treating some streams as non-gRPC. The fact that rolling back policy resolves the issue points to ALB-side handling of TLS1.3 policy or client ALPN negotiation problems rather than backend application errors.

Suggested Fixes

Diagnostic step 1

Collect ALB access logs and ELB TLS logs with timestamps for failed requests and correlate with backend logs; capture full ALB flow tracing if using AWS X-Ray or ALB request tracing.

Diagnostic step 2

Use a controlled test client that supports TLS1.3 and ALPN h2 to reproduce errors; record client TLS handshake and ALPN negotiation (e.g. using openssl s_client -alpn h2 -tls1_3) against the ALB to compare successful vs failing traces.

Diagnostic step 3

Test a narrow change: create a separate ALB listener using TLS1.3-only policy and point a small percentage of traffic or a canary client there to replicate without impacting majority of traffic.