1
Diagnostic step 1
Short-term mitigation: increase system and PM2 worker ulimit (fs.file-max and limits.conf) and restart processes in a rolling fashion. Monitor FD usage with lsof and /proc/<pid>/fd counts. This is reversible and low-risk.
FULL AI ANALYSIS
Next.js API processes hit 'Too many open files' under load behind PM2 — 500s only in production
Generated Oct 14, 2026, 9:39:02 AM
Analysis summary
EMFILE originates from exhausting per-process file descriptor limits. Evidence (ulimit 1024, many TIME_WAIT sockets, temp file usage, synchronous file ops) indicates a mix of high concurrent socket counts and file handle usage per worker. Increasing ulimit reduced incidents, confirming FD exhaustion. Root causes likely a combination of low OS limits, synchronous file handling that blocks and keeps handles open, and insufficient connection pooling leading to many ephemeral sockets. Resource leak in application less likely because raised ulimit improved behavior and targeted code changes helped.
Possible causes
Operating system per-process file descriptor limit too low for PM2 worker concurrency combined with high concurrent upstream sockets and temp-file usage
80%Application synchronous file operations and temp-file lifecycle holding descriptors open longer than needed (blocking under load)
65%Insufficient reuse/connection pooling for upstream services leading to many short-lived sockets (TIME_WAIT) per worker
50%File descriptor leak in third-party native module (less likely given ulimit change impact)
25%Suggested fixes
2
Diagnostic step 2
Instrument the app to record open descriptor counts on each worker periodically and log stack traces or endpoint context when descriptor count passes thresholds.
3
Diagnostic step 3
Replace blocking fs operations in hot paths with streaming async APIs; ensure temp files are created with secure atomic temp file libraries and closed promptly, or use in-memory streams if payload size permits.
4
Diagnostic step 4
Review and enable connection pooling to upstream services (DB, auth) and tune TCP TIME_WAIT handling (reuse via SO_REUSEPORT not recommended lightly); consider reusing keep-alive sockets.
5
Diagnostic step 5
If FD leaks suspected from native modules, run a controlled canary with heap/FD tracing and isolate suspected modules. Escalate to module maintainers or Node.js support if a native bug is reproducible with small repro.