Next.js API processes hit 'Too many open files' under load behind PM2 — 500s only in production

Environment

Next.js 13 app deployed on Ubuntu 22.04 VMs managed by PM2 cluster mode (4 worker processes). Production uses proxy (NGINX) and networked log aggregation. Load test and production peaks observed — error rate rises only in prod under sustained traffic bursts.

Symptoms

Under high concurrency, API routes intermittently return HTTP 500 and PM2 worker logs show 'Error: EMFILE: too many open files' stack traces referencing fs.readFileSync and a custom temp-file cleanup routine. Local development and staging (lower concurrency) do not show the issue.

Observations

ulimit -n on host is 1024. PM2 spawns 4 workers; each worker appears to open many file descriptors (sockets to upstream auth service, DB connections, file handles). lsof during peak shows many file descriptors in TIME_WAIT and open file handles to /tmp/app-*.tmp. The app uses synchronous file writes/reads for multipart processing and relies on process.on('exit') for cleanup. Application code occasionally leaves temp files if process crashes.

Already attempted

Increased ulimit for one test VM to 65536 and restarted PM2 — error frequency dropped but not eliminated. Replaced synchronous fs.readFileSync with async streaming in a few critical handlers in a canary build — those endpoints improved. Rotated temp file cleanup to periodic cron but didn't deploy cluster-wide yet. No evidence of FD leak in native modules.

Question

What's the best approach to identify and fix the cause of EMFILE in production Next.js behind PM2, and what immediate production mitigations are safe to apply without code rollbacks?

Post timeline

COMMUNITY RESPONSES

Verified community troubleshooting

0 replies

No answers yet..

Showing 0 of 0 community responses

AI DIAGNOSTICS

Incident analysis

Updated 1d ago• Trouble ID

Most likely causes

Operating system per-process file descriptor limit too low for PM2 worker concurrency combined with high concurrent upstream sockets and temp-file usage 80%

Application synchronous file operations and temp-file lifecycle holding descriptors open longer than needed (blocking under load) 65%

Insufficient reuse/connection pooling for upstream services leading to many short-lived sockets (TIME_WAIT) per worker 50%

Summary

EMFILE originates from exhausting per-process file descriptor limits. Evidence (ulimit 1024, many TIME_WAIT sockets, temp file usage, synchronous file ops) indicates a mix of high concurrent socket counts and file handle usage per worker. Increasing ulimit reduced incidents, confirming FD exhaustion. Root causes likely a combination of low OS limits, synchronous file handling that blocks and keeps handles open, and insufficient connection pooling leading to many ephemeral sockets. Resource leak in application less likely because raised ulimit improved behavior and targeted code changes helped.

Suggested Fixes

Diagnostic step 1

Short-term mitigation: increase system and PM2 worker ulimit (fs.file-max and limits.conf) and restart processes in a rolling fashion. Monitor FD usage with lsof and /proc/<pid>/fd counts. This is reversible and low-risk.

Diagnostic step 2

Instrument the app to record open descriptor counts on each worker periodically and log stack traces or endpoint context when descriptor count passes thresholds.

Diagnostic step 3

Replace blocking fs operations in hot paths with streaming async APIs; ensure temp files are created with secure atomic temp file libraries and closed promptly, or use in-memory streams if payload size permits.