FULL AI ANALYSIS

Systemd unit hangs systemd-shutdown when TLS cert-rotation script runs in ExecStop on Ubuntu 22.04

Generated Oct 3, 2026, 12:35:03 AM

Analysis summary

Long-running networked operations in ExecStop are brittle because shutdown can close network interfaces, DNS, or firewall rules before the operation completes; systemd expects units to stop promptly. Running blocking external calls during shutdown risks hanging the init flow. The right approach is to make stop actions fast and idempotent, offloading long work to background jobs with proper timeouts and checkpoints.

Possible causes

ExecStop runs network-dependent commands without timeouts; network stack may be partially torn down during shutdown causing the script to block indefinitely.

95%

Service design couples certificate rotation (long task) to process termination instead of using an asynchronous mechanism or pre-shutdown hook.

90%

Default systemd stop timeout is insufficient for network delays, leading to forceful termination or hangs if the script ignores SIGTERM.

80%

Child processes spawned by the script are not properly reaped, and systemd waits on the main process which itself blocks.

75%

Suggested fixes

Diagnostic step 1

Refactor the workflow: remove long-running network calls from ExecStop. Instead, have ExecStop trigger a short, idempotent marker (e.g., create a file or send a systemd message) and let a separate timer/service perform the rotation asynchronously (systemd oneshot timer).

Diagnostic step 2

If rotation must be attempted during shutdown, add explicit, conservative timeouts in the script for network operations and ensure the script handles SIGTERM to exit promptly; set TimeoutStopSec to a controlled value and document it.

Diagnostic step 3

Use systemd notify or transient units: ExecStop can start a systemd-run --unit=cert-rotate --on-active=0 --collect to run rotation detached from shutdown so shutdown proceeds while rotation completes in parallel (reversible configuration).

Diagnostic step 4

Instrument the script with robust logging and ensure child processes are started in a way that systemd can track or that they are reparented safely; avoid backgrounding without systemd supervision.

Diagnostic step 5

If rotation requires reliable completion before shutdown (rare), consider scheduling maintenance windows instead of relying on shutdown-triggered rotation; escalate to platform architect if policy requires rotation on stop.

Contact • Help • API Docs • Careers • Terms • Privacy • Cookies • Ads