1
Diagnostic step 1
Check process limits: run 'cat /proc/$(pidof server)/limits' and increase nofile (systemd LimitNOFILE) temporarily to a high value (e.g., 65k) to see if errors drop.
FULL AI ANALYSIS
Go gRPC server returns HTTP/2 GOAWAY under high concurrent RPCs — client sees stream resets
Generated Oct 13, 2026, 9:45:52 AM
Analysis summary
GOAWAY/stream resets under heavy load commonly stem from HTTP/2 flow-control or concurrency limits (MaxConcurrentStreams) on server or intermediary, gRPC internal resource exhaustion (goroutine/connection pool limits), or upstream load-balancer TCP behaviors (connection churn, proxy-level limits). Since the server runs behind an NLB and GOAWAY appears in server logs, focus on HTTP/2 settings, gRPC runtime limits, and OS-level file/socket limits. Reproducible locally under load suggests server-side limits rather than client bugs.
Possible causes
Server hitting HTTP/2 MaxConcurrentStreams limit; gRPC issues GOAWAY when per-connection streams exceed configured limits.
75%OS file descriptor limits (ulimit) reached causing accept failures and connection resets under peak concurrency.
70%gRPC server running out of available goroutines or heap/stack contention causing internal timeouts and shutdown of connections.
35%AWS NLB TCP connection churn or idle timeouts causing clients to reconnect and trigger GOAWAY handling edge cases.
15%Suggested fixes
2
Diagnostic step 2
Enable gRPC verbose logging (GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=99) to collect detailed HTTP/2/stream events; rotate logs to avoid disk fill.
3
Diagnostic step 3
Expose current HTTP/2 MaxConcurrentStreams: instrument server to log http2 server settings or use Wireshark/tcpdump on localhost to inspect SETTINGS frames exchanged during handshake.
4
Diagnostic step 4
Run a controlled load test against the server directly (bypass NLB) to confirm if GOAWAY still appears — if not, inspect NLB connection reuse/idle-timeout and consider enabling proxy-protocol or switching to ALB that preserves HTTP/2 if required.
5
Diagnostic step 5
Temporarily raise gRPC server-side concurrency settings (e.g., grpc.MaxConcurrentStreams via ServerOptions) and observe behavior; rollback if memory/goroutine usage increases beyond safe thresholds. Monitor memory/CPU with pprof and goroutine counts (runtime.NumGoroutine).