PgBouncer reports 'server closed the connection unexpectedly' during peak write bursts

Environment

Production web service using PostgreSQL 13 primary on a dedicated VM and PgBouncer 1.17 in transaction-pooling mode running in a Docker container on same host as application. Peak load: 600 conn/s with many short write transactions; PgBouncer max_client_conn set to 5000; server_max_conn to 200. Docker container has ulimit defaults.

Symptoms

Under peak write bursts, applications intermittently see errors: 'server closed the connection unexpectedly' and transactions abort. PgBouncer logs show repeated 'server_error' entries and 'server disconnected' lines. The PostgreSQL server logs show idle connections being terminated with 'terminating connection due to administrator command' occasionally and OOM killer not invoked.

Observations

When the issue occurs, pg_stat_activity on the primary shows many connections in 'idle in transaction' state briefly then disappearing. Docker container running PgBouncer shows 'Too many open files' errors in PgBouncer logs during reproductions. Increasing file descriptor limits at the container runtime (ulimit -n) reduces the frequency of errors but does not eliminate them. Network latency between application and PgBouncer is negligible (localhost).

Already attempted

Raised PgBouncer's 'max_client_conn' and 'reserve_pool_size'; adjusted server_max_conn to better match PostgreSQL's max_connections; restarted PgBouncer to clear state. Tuned pool_mode to statement briefly for testing (potentially unsafe for application logic) and observed fewer errors but different transactional behavior. Monitored host fd usage with lsof and found PgBouncer approaching OS limits during bursts.

Question

How can I determine whether PgBouncer 'server closed the connection unexpectedly' is caused by file descriptor exhaustion in the container, server-side max_connections limits, or a misconfiguration of pooling mode under heavy write load, and what reversible mitigation steps will stabilize connections during peak bursts?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1w ago• Trouble ID

Most likely causes

File descriptor limit in the Docker container prevents PgBouncer from opening the required number of sockets to backend servers and client connections under peaks. 90%

Pooling mode (transaction) leads to many short-lived server connections during bursty writes; statement pooling temporarily masked the issue but may break transactional semantics. 70%

PgBouncer configured with inadequate server_max_conn relative to client demand causes rapid churn of server connections and stress on OS resources. 55%

Summary

Logs and observations point primarily to resource exhaustion at the PgBouncer process (open file descriptors) compounded by pooling configuration mismatches for a write-heavy workload. The 'Too many open files' messages and improvement after raising ulimits indicate the container's default limits are insufficient. Additionally, transaction pooling modes (transaction vs statement) and server max_connections interplay can cause PgBouncer to aggressively open/close server sockets, exacerbating fd usage.

Suggested Fixes

Diagnostic step 1

Increase the container and PgBouncer process file descriptor limit (set ulimit -n at container start and configure systemd/docker service limits) and monitor for a reduction in 'Too many open files' errors (reversible configuration change).

Diagnostic step 2

Align PgBouncer's server_max_conn with PostgreSQL's max_connections and ensure there is headroom for other tools; avoid setting server_max_conn higher than the database can accept.

Diagnostic step 3

Enable PgBouncer's verbose logging temporarily and capture metrics (show pools, SHOW STATS) during a controlled load test to correlate fd usage with connection churn.