Go gRPC server returns HTTP/2 GOAWAY under high concurrent RPCs — client sees stream resets

Environment

Go 1.20 gRPC server (google.golang.org/grpc v1.55), running on Debian 12 VMs behind AWS NLB; clients are varied (Go, Java) making 1000+ concurrent unary RPCs per second to the same server instance.

Symptoms

Clients intermittently receive HTTP/2 GOAWAY frames and STREAM_CLOSED errors mid-request under peak concurrency. Error rate spikes correlate with traffic bursts; successful RPCs complete normally when concurrency reduced.

Observations

Server logs show repeated 'transport: http2Server.HandleStreams failed' followed by connection close. netstat shows many TIME_WAIT connections. Strace on server process shows bursts of accept/read syscalls, no obvious user-space panic. Increasing server's MaxConcurrentStreams at application layer not previously configured.

Already attempted

Restarted server process (temporary relief), increased Go runtime GOMAXPROCS to match CPU cores, bumped keepalive settings, moved NLB healthcheck from TCP to HTTP, ran load tests locally reproducing GOAWAY only under high concurrency. Did not change application code.

Question

What are the likely causes of GOAWAY under high concurrency in this Go gRPC deployment, and what safe diagnostics and mitigations should I run to narrow root cause and restore stable RPC throughput?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 2d ago• Trouble ID

Most likely causes

Server hitting HTTP/2 MaxConcurrentStreams limit; gRPC issues GOAWAY when per-connection streams exceed configured limits. 75%

OS file descriptor limits (ulimit) reached causing accept failures and connection resets under peak concurrency. 70%

gRPC server running out of available goroutines or heap/stack contention causing internal timeouts and shutdown of connections. 35%

Summary

GOAWAY/stream resets under heavy load commonly stem from HTTP/2 flow-control or concurrency limits (MaxConcurrentStreams) on server or intermediary, gRPC internal resource exhaustion (goroutine/connection pool limits), or upstream load-balancer TCP behaviors (connection churn, proxy-level limits). Since the server runs behind an NLB and GOAWAY appears in server logs, focus on HTTP/2 settings, gRPC runtime limits, and OS-level file/socket limits. Reproducible locally under load suggests server-side limits rather than client bugs.

Suggested Fixes

Diagnostic step 1

Check process limits: run 'cat /proc/$(pidof server)/limits' and increase nofile (systemd LimitNOFILE) temporarily to a high value (e.g., 65k) to see if errors drop.

Diagnostic step 2

Enable gRPC verbose logging (GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=99) to collect detailed HTTP/2/stream events; rotate logs to avoid disk fill.

Diagnostic step 3

Expose current HTTP/2 MaxConcurrentStreams: instrument server to log http2 server settings or use Wireshark/tcpdump on localhost to inspect SETTINGS frames exchanged during handshake.